Laguna-S 2.1 NVFP4

poolside's agentic-coding Laguna-S 2.1 (118B MoE, ~8B active) in NVFP4 on one DGX Spark, with the matching NVFP4 DFlash speculator — 42.9 tok/s single-stream (2.3x the no-draft baseline) at the full 262K context.

Contributors

@Fogle

Laguna-S 2.1 is poolside's 118B agentic-coding MoE (256 routed experts + 1 shared, 10 active per token, ~8B active) with a 262K context and hybrid attention (1 full-attention layer in 4, the rest sliding-window-512). Its NVFP4 checkpoint is 71.9 GB — small enough for a single 121 GiB DGX Spark, which is the reason to pick this build over the two-Spark FP8 one: it frees the second Spark entirely, and a single node avoids the cross-node CUDA-graph deadlock that forces the TP2 recipe onto --enforce-eager, so this serves with graphs. Two things shape the config. First, the speed lever is the DFlash speculator, not the quant: the checkpoint leaves attention in BF16 (the dominant per-token read), so NVFP4 vs FP8 barely changes decode — but the draft turns 18.9 tok/s into 42.9 (2.3x, greedy so lossless). Second, and the trap worth knowing: DFlash drafts are TARGET-PRECISION-SPECIFIC. A draft ships no lm_head of its own — it shares the target's and is calibrated to the target's hidden states — so it only works with the target it was made for. You must pair this NVFP4 target with poolside's -DFlash-NVFP4 draft (61.9% acceptance here); the -DFlash-FP8 draft (or a W4A16 quant of it) gives 0% acceptance against an NVFP4 target and drops decode below the no-draft baseline. Memory is a non-fight: hybrid attention makes the full 262K context fit ~1.8x over, so we pin the KV pool down to 1.25x and leave the rest free.

• 1 × DGX Spark (GB10, sm_121) — the whole 118B model in NVFP4, ~72 GB of weights, second Spark left free
• DFlash speculative decode at k=6: 42.9 tok/s single-stream on code vs 18.9 no-draft — 2.3x
• --max-num-seqs 4 (not the reflexive 1-2): under spec-decode it sets the CUDA-graph capture size, worth a measured +7% single-stream (43 vs 40 tok/s at k=6)
• 61.9% draft acceptance at k=6, 3.71 tokens/step (per-position 0.88 / 0.69 / 0.64 / 0.59 / 0.51 / 0.39)
• Greedy (temp 0), so spec output is token-identical to the baseline — the speedup is free of any quality cost
• Serves with CUDA graphs (no --enforce-eager) — a single node has no cross-node cudagraph deadlock
• MUST pair the NVFP4 target with the -DFlash-NVFP4 draft: DFlash drafts are target-precision-specific (the -DFlash-FP8/W4A16 draft gives 0% acceptance here)
• MoE backend auto-selects FLASHINFER_CUTLASS for NVFP4 on sm_121 — no --moe-backend flag needed
• Full 262,144-token context serves; KV pinned to a 327K-token pool (1.25x), ~24 GiB free
• Slower than the two-Spark FP8 build (60.9 tok/s) — one Spark's ~273 GB/s vs two summed — but frees a whole node
• First boot JITs sm_121a kernels + compiles graphs for ~15-20 min (cached after; looks like a hang, it isn't)

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.25.1 + torch 2.11.0+cu130 (~/venvs/vllm-025). LagunaForCausalLM and DFlashLagunaForCausalLM are natively registered — no --trust-remote-code
• CUDA 13 is required for the FP4 MoE kernels (a cu129 build fails). flashinfer 0.6.13 ships no prebuilt sm_121a cubins — budget one ~15-20 min JIT + torch.compile on the first boot
• ~75 GB free disk for the checkpoint + draft
• Base model: poolside/Laguna-S-2.1 (license OpenMDW-1.1)

Quick start

1. 1

   Free the node, then stage the NVFP4 checkpoint and its matching draft

   Single node, so no fabric mirroring. Purge the page cache first — vLLM's startup check wants free memory >= util*total, and a cold box with page cache trips it. Pull the NVFP4 weights AND the NVFP4-matched DFlash draft (not the FP8 draft — see the warning in troubleshooting).

   pkill -f '[v]llm serve' 2>/dev/null
   python3 ~/Dev/vLLM-Moet/spark/purge-cache.py ~/models/hf
   
   hf download poolside/Laguna-S-2.1-NVFP4 --local-dir ~/models/hf/Laguna-S-2.1-NVFP4
   hf download poolside/Laguna-S-2.1-DFlash-NVFP4 --local-dir ~/models/hf/Laguna-S-2.1-DFlash-NVFP4bash

   If `hf download` stalls (the xet path hangs on these boxes), set HF_HUB_DISABLE_XET=1. Verify the target has its 14 safetensors shards and zero *.incomplete files.

2. 2

   Serve NVFP4 + the NVFP4 DFlash draft under a memory cap

   No Ray, no TP — a single vLLM process. --speculative-config turns on DFlash at k=6 (the measured peak). --kv-cache-memory-bytes pins the KV pool to ~1.25x for a single user instead of the idle 1.82x at util 0.85. No --moe-backend flag — vLLM auto-selects FLASHINFER_CUTLASS for the NVFP4 MoE on sm_121. No --enforce-eager — a single node runs CUDA graphs without the cross-node deadlock the two-Spark recipe hits.

   VENV=$HOME/venvs/vllm-025
   systemd-run --user --scope --collect -p MemoryMax=118G -p MemorySwapMax=0 \
     env PATH="$VENV/bin:$HOME/.local/bin:/usr/local/cuda/bin:$PATH" VLLM_USE_DEEP_GEMM=0 \
     vllm serve ~/models/hf/Laguna-S-2.1-NVFP4 \
     --served-model-name laguna-s-nvfp4 \
     --max-model-len 262144 \
     --gpu-memory-utilization 0.85 \
     --kv-cache-memory-bytes 12884901888 \
     --max-num-seqs 4 --max-num-batched-tokens 8192 \
     --speculative-config '{"method":"dflash","model":"'$HOME'/models/hf/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":6}' \
     --host 0.0.0.0 --port 8000bash

   First boot pays a ~15-20 min flashinfer sm_121a JIT + torch.compile (cached after; the log freezes and the GPU idles — it is compiling, not hung). Expect 'Using FLASHINFER_CUTLASS NvFp4 MoE backend' and 'GPU KV cache size: 327,717 tokens ... 1.25x'.

3. 3

   Smoke test, warm up, then read acceptance

   Confirm coherence, warm the kernels (spec-decode JITs extra Triton kernels on first use), then read the acceptance rate — that number is why decode is 2.1x.

   curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
     -d '{"model":"laguna-s-nvfp4","messages":[{"role":"user","content":"capital of France? one word"}],"max_tokens":8,"temperature":0}'
   # -> "Paris"
   
   # a few hundred tokens of throwaway traffic, then:
   curl -s localhost:8000/metrics | grep -E 'spec_decode_num_(draft|accepted)_tokens_total'
   # accepted / drafted ~= 0.59 on code at k=6
   free -g | sed -n '2p'   # ~94 GiB used, ~26 freebash

   Warm before benchmarking. Acceptance is content-dependent: ~59% on code at k=6; reasoning/thinking prose drafts worse (thinking is off by default in the chat template, which is right for a coding endpoint). If acceptance is 0%, you paired the wrong draft — see troubleshooting.

Key vLLM parameters

API usage

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"laguna-s-nvfp4",
       "messages":[{"role":"user","content":"Complete this module: import csv\n\ndef parse_csv(path):"}],
       "max_tokens":256, "temperature":0, "stream":true,
       "stream_options":{"include_usage":true}}'

# decode_tps = (completion_tokens - 1) / (t_last_token - t_first_token)
# measured: 42.9 tok/s at a 2048-token prompt; 61.9% acceptance, 3.71 tokens/stepbash

Troubleshooting

The DFlash draft gives 0% acceptance and decode drops below the no-draft baseline

You paired the wrong-precision draft. DFlash drafts are target-precision-specific — a draft shares the target's lm_head and is calibrated to its hidden states. This NVFP4 target needs poolside/Laguna-S-2.1-DFlash-NVFP4; the -DFlash-FP8 draft (or a W4A16 quant of it) yields 0% here (verification rejects 100%, so you pay pure draft overhead). Always check `spec_decode_num_accepted_tokens_total` > 0 after swapping a draft.

Engine refuses to start: 'estimated maximum model length is 128192' / KV cache larger than available

The KV pin is too small for a single node. On TP2 the KV is sharded so 5 GiB/rank suffices; on ONE node the whole pool is on that node, and 262K needs ~9.6 GiB. Use --kv-cache-memory-bytes 12884901888 (12 GiB) or larger.

Startup fails: 'Free memory on device cuda:0 (NN GiB) is less than desired GPU memory utilization'

Page cache or a zombie process is holding memory. Purge the page cache (`purge-cache.py ~/models/hf`) and kill any leftover vLLM/Ray from prior runs (kill by PID; `pkill -f '[v]llm'` can drop your SSH). A clean box shows ~117 GiB free.

Drafter load dies with KeyError: 'weight_packed' after switching drafts

vLLM's torch.compile cache doesn't re-key on the draft's quantization, so a graph compiled for one draft precision is reloaded for another and mismatches. Point VLLM_CACHE_ROOT at a fresh dir for the new draft, or serve --enforce-eager (no compiled graph). Only bites when A/B-ing drafts of different quant.

Server log freezes for ~15-20 min on first boot; GPU idle

flashinfer JITs sm_121a kernels (no prebuilt cubins for GB10) and torch.compile builds the graphs. Confirm with `pgrep -af 'ninja|nvcc'`. Cached afterwards — later boots are ~3-5 min. Do not kill it.

Decode is slower than the two-Spark FP8 recipe

Expected. Single-stream decode is bandwidth-bound and one Spark has ~273 GB/s; the two-Spark FP8 build sums two nodes' bandwidth (~26.5 tok/s no-draft vs 18.9 here). This recipe trades peak decode for using only one node — pick it when you want the second Spark free, not when you want maximum tok/s.

A published single-node figure (e.g. '77 tok/s' at k=15) that you can't reproduce

Reproduce before believing it. One such '77' was a typo for 7, at num_speculative_tokens=15 — and k=15 measures 42.1 tok/s / 31% acceptance here, below the k=6 peak (43-44 tok/s / 62%). Large k craters DFlash acceptance on this model; k=6 is the peak. Sweep k yourself and read spec_decode acceptance from /metrics.