Laguna-S 2.1 FP8

poolside's agentic-coding Laguna-S 2.1 (118B MoE, ~8B active) in FP8 on two DGX Sparks (TP2), with a self-quantized W4A16 DFlash speculator — 60.9 tok/s single-stream (2.3x the no-draft baseline) at the full 262K context.

Laguna-S 2.1 is poolside's 118B agentic-coding MoE (256 routed experts + 1 shared, 10 active per token, ~8B active) with a native 262K context via YaRN and a hybrid attention stack — 1 full-attention layer in every 4, the rest sliding-window-512. This recipe serves the FP8 build across two DGX Sparks, because at 112.7 GiB the checkpoint won't leave a single 121 GiB node any room for KV. Three things shape the config. First, TP2 over the RoCE link is required for fit, and on GB10 cross-node CUDA-graph replay deadlocks (the engine dies with 'RPC call to sample_tokens timed out', head GPU pinned at 96% while the worker sits idle), so the serve runs --enforce-eager. Second, memory is a non-fight: the sliding-window attention makes the full 262K context fit ~13x over, so instead of climbing toward more context we pin the KV pool down to ~1x and hand ~37 GiB/rank back. Third, and where the speed comes from: the checkpoint leaves attention in BF16, so a smaller expert quant does NOT make decode faster — the lever is the DFlash speculator. poolside ships one DFlash draft per target precision (they share the target's lm_head, so they don't cross precisions); the FP8-target draft works (52% acceptance, 58.4 tok/s at k=8), and pairing the NVFP4-target draft with this FP8 target gives 0% (see troubleshooting). So we quantized the FP8 draft ourselves to W4A16 (published as sapidlabs/Sparkulator-Laguna-S-2.1): 2.5x smaller and, because a DFlash draft runs on every decode step, ~4% faster decode at identical acceptance — 60.9 tok/s single-stream on code at k=6, 2.3x the 26.5 tok/s no-draft baseline. Greedy verification makes the draft lossless, so the speedup costs nothing in output quality.

• 2 × DGX Spark (GB10, sm_121), tensor-parallel (TP2) — the full 118B model, FP8, ~56.6 GB of weights per node
• W4A16 DFlash speculative decode at k=6: 60.9 tok/s single-stream on code vs 26.5 no-draft — 2.3x
• Our W4A16 draft beats poolside's own FP8 draft (58.4 tok/s) by ~4% at identical 52% acceptance, and is 2.5x smaller (2.08 → 0.82 GiB)
• 62.2% draft acceptance at k=6, 3.73 tokens/step (per-position 0.89 / 0.76 / 0.65 / 0.55 / 0.48 / 0.41)
• Greedy (temp 0), so spec output is token-identical to the baseline — the speedup is free of any quality cost
• Full 262,144-token context actually serves; KV pinned to a 273K-token pool (1.04x) and ~37 GiB/rank handed back
• --enforce-eager is required: cross-node TP2 cudagraphs deadlock on GB10 (RPC sample_tokens timeout)
• DFlash drafts are target-specific: pair the FP8-family draft with this FP8 target (poolside's -DFlash-NVFP4 is for the NVFP4 checkpoint and gives 0% acceptance here)
• Decode measured at a 2048-token code prompt, warm; TTFT grows from 0.7s at 2K to 8.9s at 32K, 47s at 128K, 118s at 248K
• Two vLLM patches for the quantized draft (get_cache_scale guard + regex ignore) — see troubleshooting

Software requirements

• 2 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.25.1 + torch 2.11.0+cu130 in a venv (~/venvs/vllm-025). LagunaForCausalLM and DFlashLagunaForCausalLM are natively registered — no --trust-remote-code
• A vLLM patched so a quantized DFlash draft loads: guard get_cache_scale in laguna_dflash.py (one line, both nodes — see the patch step)
• loginctl enable-linger on BOTH nodes — without it the TP2 worker raylet dies on every SSH disconnect (systemd --user manager torn down)
• Direct RoCE link with static point-to-point IPs; resolve each node's RoCEv2 GID index at start time (they differ per node and drift)
• ~122 GB free disk per node — both nodes need the full checkpoint + draft at the same path
• Passwordless SSH between nodes. Base model: poolside/Laguna-S-2.1 (license OpenMDW-1.1)

Quick start

1. 1

   Enable user lingering + free the nodes (preflight, once per box)

   TP2 places ranks with Ray, and the raylets run as systemd --user services. Without lingering, the user manager (and every raylet) is torn down the moment the last SSH session closes — which any 'ssh spark purge-cache' inside a serve script does, so the worker silently drops and vLLM hangs forever in 'Waiting for creating a placement group'. Enable lingering once, then start from a clean slate.

   # once per box — the fix for the worker raylet dying on SSH disconnect
   ssh 192.168.100.1 'loginctl enable-linger $USER'
   ssh 192.168.100.2 'loginctl enable-linger $USER'
   
   # clean slate before serving
   pkill -f '[v]llm serve' 2>/dev/null
   python3 ~/Dev/vLLM-Moet/spark/purge-cache.py ~/models/hf
   ssh 192.168.100.2 'python3 ~/Dev/vLLM-Moet/spark/purge-cache.py ~/models/hf'bash

   Verify with `loginctl show-user $USER | grep Linger` → Linger=yes on both nodes.

2. 2

   Stage the FP8 checkpoint and the W4A16 draft on both nodes

   Each TP rank reads all shards, so both nodes need the complete tree at an identical path. Download the 121 GB FP8 checkpoint once and mirror it over the direct RoCE link (far faster than pulling twice). Pull the W4A16 DFlash draft from the Hub — or build it yourself from poolside's FP8 draft (next note).

   # node 1
   hf download poolside/Laguna-S-2.1-FP8 --local-dir ~/models/hf/Laguna-S-2.1-FP8
   hf download sapidlabs/Sparkulator-Laguna-S-2.1 --local-dir ~/models/hf/Laguna-S-2.1-DFlash-W4A16
   
   # mirror both to node 2 over the direct RoCE link
   for d in Laguna-S-2.1-FP8 Laguna-S-2.1-DFlash-W4A16; do
     rsync -a --partial --inplace \
       -e 'ssh -c aes128-gcm@openssh.com -o Compression=no' \
       ~/models/hf/$d/ 192.168.100.2:~/models/hf/$d/
   donebash

   To rebuild the W4A16 draft yourself from poolside/Laguna-S-2.1-DFlash-FP8: `quantize_raw_w4a16.py --targets 'layers\.\d+\.self_attn\.o_proj' 'layers\.\d+\.mlp\.(gate_proj|up_proj|down_proj)' 'fc' --group-size 128`, then set the config's `ignore` to `re:` regexes (`re:.*\.self_attn\.qkv_proj$`, `re:.*\.self_attn\.g_proj$`). Leave qkv_proj BF16 — see troubleshooting. Verify both nodes have 24 FP8 shards and zero *.incomplete files.

3. 3

   Patch vLLM so the quantized DFlash draft loads (both nodes)

   vLLM 0.25.1's Laguna DFlash loader assumes an unquantized (or FP8) draft: it calls quant_config.get_cache_scale(), an FP8-only method that CompressedTensorsConfig (W4A16) doesn't have, and dies with AttributeError before it ever serves. Guard the call so a compressed-tensors draft falls through to the generic scale path. Apply on BOTH nodes — each has its own site-packages.

   # vllm/model_executor/models/laguna_dflash.py — DFlashLagunaModel.load_weights
   -            if self.quant_config is not None and (
   +            if self.quant_config is not None and hasattr(
   +                self.quant_config, "get_cache_scale"
   +            ) and (
                    scale_name := self.quant_config.get_cache_scale(name)
                ):python

   The W4A16 draft has no KV-cache scales, so returning no cache-scale name is correct. The other half of making a quantized draft load is the checkpoint's `ignore` list using `re:` regexes (done at quant time) so vLLM leaves qkv_proj/g_proj unquantized — literal names don't match vLLM's prefixed module names and the layer gets wrongly quantized, then KeyErrors on its .weight.

4. 4

   Bring up the 2-node Ray cluster (raylets as detached services)

   Start each raylet as a systemd --user *service* (not --scope): pass the fabric env via --setenv, raise LimitMEMLOCK for RDMA, and put ninja + nvcc on PATH (the FP8 MoE and the draft kernels JIT on the first forward and die with 'No such file or directory: ninja' otherwise). Resolve each node's RoCEv2 GID index at start time — they differ per node and drift across reboots.

   VENV=$HOME/venvs/vllm-025
   ENV='--setenv=PATH='$HOME'/venvs/vllm-025/bin:'$HOME'/.local/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
     --setenv=NCCL_IB_DISABLE=0 --setenv=NCCL_IB_HCA=rocep1s0f1 \
     --setenv=NCCL_SOCKET_IFNAME=enp1s0f1np1 --setenv=GLOO_SOCKET_IFNAME=enp1s0f1np1 \
     --setenv=RAY_memory_monitor_refresh_ms=0'
   
   gid() { ssh $1 "for g in /sys/class/infiniband/rocep1s0f1/ports/1/gids/*; do i=\${g##*/}; \
     [ \"\$(cat /sys/class/infiniband/rocep1s0f1/ports/1/gid_attrs/types/\$i 2>/dev/null)\" = 'RoCE v2' ] || continue; \
     case \$(cat \$g) in *ffff:$(printf '%02x%02x:%02x%02x' $(echo $2|tr . ' ')) ) echo \$i; break;; esac; done"; }
   GID1=$(gid 192.168.100.1 192.168.100.1); GID2=$(gid 192.168.100.2 192.168.100.2)
   
   ssh 192.168.100.1 "systemd-run --user --collect --unit=laguna-ray-head $ENV \
     --setenv=VLLM_HOST_IP=192.168.100.1 --setenv=NCCL_IB_GID_INDEX=$GID1 \
     -p MemoryMax=98G -p MemorySwapMax=0 -p LimitMEMLOCK=infinity \
     $VENV/bin/ray start --block --head --node-ip-address=192.168.100.1 --port=6379 --object-store-memory=2000000000"
   sleep 8
   ssh 192.168.100.2 "systemd-run --user --collect --unit=laguna-ray-worker $ENV \
     --setenv=VLLM_HOST_IP=192.168.100.2 --setenv=NCCL_IB_GID_INDEX=$GID2 \
     -p MemoryMax=110G -p MemorySwapMax=0 -p LimitMEMLOCK=infinity \
     $VENV/bin/ray start --block --address=192.168.100.1:6379 --object-store-memory=2000000000 --node-ip-address=192.168.100.2"
   sleep 10
   $VENV/bin/ray status   # expect 2 nodes, 2.0 GPUbash

   --block keeps `ray start` as the service's main process so the unit stays active. If `ray status` shows 1.0 GPU, the worker dropped — check lingering is enabled and that the worker service is `active` (systemctl --user is-active laguna-ray-worker on .2).

5. 5

   Serve TP2 + W4A16 DFlash under a memory cap

   --enforce-eager is load-bearing: cross-node cudagraph replay deadlocks on GB10. --speculative-config turns on DFlash at k=6 (the measured peak; sweep in keyParams). --kv-cache-memory-bytes pins the KV pool to ~1x for a single user instead of the idle 12.79x at util 0.85. The driver runs in a systemd scope with MemorySwapMax=0 so any surprise is a clean kill, not a swap-wedge.

   VENV=$HOME/venvs/vllm-025
   export VLLM_HOST_IP=192.168.100.1
   export NCCL_SOCKET_IFNAME=enp1s0f1np1 GLOO_SOCKET_IFNAME=enp1s0f1np1
   export NCCL_IB_DISABLE=0 NCCL_IB_HCA=rocep1s0f1 NCCL_IB_GID_INDEX=$GID1
   export RAY_memory_monitor_refresh_ms=0
   
   systemd-run --user --scope --collect -p MemoryMax=12G -p MemorySwapMax=0 \
     $VENV/bin/vllm serve ~/models/hf/Laguna-S-2.1-FP8 \
     --served-model-name laguna-s-fp8 \
     --distributed-executor-backend ray --tensor-parallel-size 2 \
     --enforce-eager \
     --max-model-len 262144 \
     --gpu-memory-utilization 0.85 \
     --kv-cache-memory-bytes 5368709120 \
     --max-num-seqs 2 --max-num-batched-tokens 8192 \
     --speculative-config '{"method":"dflash","model":"'$HOME'/models/hf/Laguna-S-2.1-DFlash-W4A16","num_speculative_tokens":6}' \
     --host 0.0.0.0 --port 8000bash

   First start loads 24 shards on each rank (~3-6 min) then the FP8 MoE + draft kernels JIT once. Expect the log to confirm 'Using MarlinLinearKernel for CompressedTensorsWNA16' (the W4A16 draft) and 'GPU KV cache size: 273,095 tokens ... 1.04x'.

6. 6

   Smoke test, warm up, then read the pool and acceptance

   Confirm coherence, warm the kernels (spec-decode JITs extra Triton kernels on first use), then read the acceptance rate — that number is why decode is 2.3x. Read steady-state memory only once the API is up.

   curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
     -d '{"model":"laguna-s-fp8","messages":[{"role":"user","content":"capital of France? one word"}],"max_tokens":8,"temperature":0}'
   # -> "Paris"
   
   # a few hundred tokens of throwaway traffic to warm kernels, then:
   curl -s localhost:8000/metrics | grep -E 'spec_decode_num_(draft|accepted)_tokens_total'
   # accepted / drafted ~= 0.62 on code at k=6
   
   free -g | sed -n '2p'                       # head ~81 GiB used
   ssh 192.168.100.2 "free -g | sed -n '2p'"   # worker ~74 GiB usedbash

   Warm before benchmarking — a cold pass faults expert planes from NVMe and JITs spec kernels, and can invert a tuning conclusion. Acceptance is content-dependent: ~62% on code at k=6; reasoning/thinking prose drafts worse (thinking is off by default in the chat template, which is right for a coding endpoint).

Key vLLM parameters

API usage

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"laguna-s-fp8",
       "messages":[{"role":"user","content":"Complete this module: import csv\n\ndef parse_csv(path):"}],
       "max_tokens":256, "temperature":0, "stream":true,
       "stream_options":{"include_usage":true}}'

# decode_tps = (completion_tokens - 1) / (t_last_token - t_first_token)
curl -s localhost:8000/metrics | grep spec_decode_num_accepted_tokens_total
# measured: 60.9 tok/s at a 2048-token prompt; 62.2% acceptance, 3.73 tokens/stepbash

Troubleshooting

vLLM hangs forever in 'Waiting for creating a placement group', only 1.0 GPU in `ray status`

The worker raylet died when an SSH session closed, because user lingering is disabled — the systemd --user manager (and the raylet) is torn down with the last session. Run `loginctl enable-linger $USER` on BOTH nodes, run the raylets as detached --user services, and confirm `systemctl --user is-active laguna-ray-worker` on .2. This cost us an hour on first bring-up.

Engine dies during the first generation: 'TimeoutError: RPC call to sample_tokens timed out'

Cross-node TP2 CUDA-graph replay deadlock on GB10 — the head GPU pins at 96% while the worker sits at 0%. NCCL_GRAPH_MIXING_SUPPORT=1 on the raylets is not enough. Serve with --enforce-eager.

Drafter fails to load: "'CompressedTensorsConfig' object has no attribute 'get_cache_scale'"

vLLM's Laguna DFlash loader assumes an FP8/unquantized draft. Guard the call in laguna_dflash.py: `... and hasattr(self.quant_config, 'get_cache_scale') and (scale_name := ...)`. Apply on every node. This only shows up with a quantized (W4A16) draft.

Drafter load KeyError: 'layers.0.self_attn.g_proj.weight' (or the layer is quantized when it shouldn't be)

The checkpoint's quantization_config `ignore` list used literal names, which vLLM matches against the PREFIXED module name (model.layers.…) and misses, so the layer gets quantized and then can't find its plain .weight. Write ignore entries as `re:` regexes, e.g. `re:.*\.self_attn\.g_proj$`. quantize_raw_w4a16.py emits literal names by default — rewrite them after quantizing.

Drafter load AttributeError: 'QKVParallelLinear' object has no attribute 'weight'

DFlash precomputes a context-KV buffer by slicing qkv_proj.weight directly (no dequant path), which a W4A16 qkv_proj doesn't have. Leave qkv_proj (and the tiny g_proj gate) in BF16 in the quant — the draft is still 2.5x smaller with only the MLP, o_proj and fc quantized — or patch a dequant into _build_context_kv_buffers.

Swapping to poolside's -DFlash-NVFP4 draft → 0% acceptance, decode drops BELOW the no-draft baseline

DFlash drafts are TARGET-PRECISION-SPECIFIC — a draft shares the target's lm_head and is calibrated to the target's hidden states, so it only works with the matching target. The -DFlash-FP8 draft (and our W4A16 quant of it) pairs with THIS FP8 target; poolside's -DFlash-NVFP4 draft is for the NVFP4 checkpoint and gives 0% acceptance here (verification rejects 100%, so you pay pure draft overhead). It is not broken — it is mismatched. Always check `spec_decode_num_accepted_tokens_total` > 0 after swapping a draft.

A very long prompt takes minutes before the first token

That is prefill, and it is honest: decode stays fast (48-66 tok/s) but TTFT grows with prompt length — measured 0.7s at 2K, 8.9s at 32K, 47s at 128K, 118s at 248K on this config. The full 262K context genuinely serves; it just costs compute to ingest.

pkill takes down your own SSH session

`pkill -f 'vllm serve'` over SSH matches the remote command's own argv. Use a character class: `pkill -f '[v]llm serve'`.