Laguna-XS 2.1 NVFP4

Serve poolside's agentic-coding Laguna-XS 2.1 (33B MoE, ~3B active) on one DGX Spark with the NVFP4 experts and the checkpoint's own DFlash speculator: 132 tok/s single-stream on code — 3.17x the no-draft baseline — with the full 262K context still fitting 1.5x over. Needs vLLM 0.25.1 (the release where the Laguna DFlash draft landed) and one non-obvious tuning call: k=6, not k=1.

Contributors

@joemuller

Laguna-XS 2.1 is poolside's 33B agentic-coding MoE (256 routed experts + 1 shared, 8 active per token, ~3B active) with a native 262K context via YaRN. This recipe serves the NVFP4 build on a single DGX Spark and leans on the one lever that actually moves single-stream decode here: the checkpoint's own DFlash speculator. Two things shape the config. First, DFlash: the Laguna draft class only landed in vLLM 0.25.1 (upstream PR #46853), and on this hardware it hits ~67% acceptance on code with num_speculative_tokens=6 — turning 41.6 tok/s into 132.2, a 3.17x single-stream win at temperature 0 (greedy, so the output is token-identical to no-draft). The best k is 6, not the k=1 you might reach for; the sweep is in the flags below. Second, memory is a non-fight: sliding-window attention makes KV so cheap that the full 262K context fits several times over, so instead of climbing toward more context we pin the KV pool down to 1.5x concurrency and hand the rest back. What you do NOT get from this checkpoint is faster decode from a smaller quant — the poolside NVFP4/FP8/INT4 builds all leave attention in BF16, which is 82% of what a token reads, so all three decode within 2 tok/s of each other. The speculator, not the quantization, is where the speed comes from. The NVFP4 quantization costs little in quality: measured in-house on HumanEval (pass@1, our own scorer since the stock lm-eval filter mis-reads this model's output), NVFP4 scores 89.0% against 91.5% for the BF16 base — a 2.5-point drop for ~3x smaller weights.

• 1 x DGX Spark (GB10, sm_121) — the whole model on one node, ~21 GB of weights
• DFlash speculative decode at k=6: 132.2 tok/s single-stream on code vs 41.6 no-draft — 3.17x
• 67.2% draft acceptance, 5.03 tokens accepted per step (per-position 0.90 / 0.81 / 0.73 / 0.62 / 0.52 / 0.46)
• Greedy (temp 0), so spec output is token-identical to the baseline — the speedup is free of any quality cost
• Quantization cost is small: 89.0% HumanEval pass@1 (NVFP4) vs 91.5% for the BF16 base, both measured in-house
• Full 262,144-token context actually serves; KV pinned to a 393K-token pool (1.5x) and ~25 GiB handed back
• vLLM 0.25.1 in a dedicated venv — the release where the Laguna DFlash draft class first works
• MoE backend auto-selects FLASHINFER_CUTLASS on sm_121 — no --moe-backend flag needed
• Decode is measured at a 2048-token code prompt, warm; TTFT grows from 1.5s at 2K to 45s at 256K
• First boot of this vLLM version JITs sm_121a kernels for ~15 min (looks like a hang — it isn't)

Software requirements

• 1 x DGX Spark (GB10, sm_121), NVIDIA driver >= 580 (CUDA-13 capable)
• vLLM 0.25.1 + torch 2.11.0+cu130. This is load-bearing: the Laguna DFlash draft (DFlashLagunaForCausalLM -> laguna_dflash) only registers in 0.25.1; on 0.24.0 it fails with 'Model architectures [DFlashLagunaForCausalLM] are not supported.' We keep it in a separate ~/venvs/vllm-025 so the 0.24.0 venv stays reproducible for older rows
• CUDA 13 is required for the FP4 MoE kernels (a cu129 build fails). If cloning a working cu130 venv, note vllm==0.25.1 pins torch==2.11.0, so pip leaves the cu130 torch in place
• flashinfer 0.6.13 (pulled in by vLLM 0.25.1) ships no prebuilt sm_121a cubins — budget one ~15-min JIT build on the first boot per venv/model-shape
• ~22 GB free disk for the checkpoint + draft. Native laguna support means no --trust-remote-code
• Base model: poolside/Laguna-XS-2.1 (license OpenMDW-1.1)

Quick start

1. 1

   Free the node first

   Unified memory means an overcommit wedges the whole box, so start from a known-clean state: kill any stale server and purge the page cache, which otherwise trips vLLM's 'free memory less than desired GPU memory utilization' check at startup.

   # the [v] keeps pkill from matching its own command line over SSH
   pkill -f '[v]llm serve' 2>/dev/null; sleep 3
   python3 ~/Dev/vLLM-Moet/spark/purge-cache.py ~/models/hf
   free -g | sed -n '2p'   # note the idle baseline (~4 GiB OS on a clean Spark)bash

   If you have a Ray cluster running from other work, its raylets sit in this baseline (~2-3 GiB); it does not affect this single-node recipe, but subtract it when reading the serving working set.

2. 2

   Stage the NVFP4 checkpoint and the DFlash draft

   One node, so no fabric mirroring. Pull both the served weights and the speculator.

   hf download poolside/Laguna-XS-2.1-NVFP4 --local-dir ~/models/hf/Laguna-XS-2.1-NVFP4
   hf download poolside/Laguna-XS-2.1-DFlash --local-dir ~/models/hf/Laguna-XS-2.1-DFlashbash

   Verify the target has its 5 safetensors shards and zero *.incomplete files before serving — a partial tree fails deep into load.

3. 3

   Serve NVFP4 + DFlash under a memory cap

   The load-bearing choices: --speculative-config turns on DFlash at k=6 (the measured peak, see keyParams); --kv-cache-memory-bytes pins the KV pool to 1.5x concurrency for a single user instead of the idle 4.64x you'd get at util 0.5; and the whole thing runs inside a systemd scope with MemorySwapMax=0 so any surprise is a clean kill, not a swap-wedge. No --moe-backend flag — vLLM auto-selects FLASHINFER_CUTLASS on sm_121. First boot pays a ~15-min flashinfer JIT (next step explains).

   systemd-run --user --scope --collect -p MemoryMax=100G -p MemorySwapMax=0 \
     env PATH="$HOME/venvs/vllm-025/bin:$PATH" VLLM_USE_DEEP_GEMM=0 \
     vllm serve ~/models/hf/Laguna-XS-2.1-NVFP4 \
     --served-model-name laguna-nvfp4 \
     --max-model-len 262144 \
     --gpu-memory-utilization 0.5 \
     --kv-cache-memory-bytes 12884901888 \
     --max-num-seqs 2 \
     --max-num-batched-tokens 8192 \
     --speculative-config '{"method":"dflash","model":"'$HOME'/models/hf/Laguna-XS-2.1-DFlash","num_speculative_tokens":6}' \
     --host 0.0.0.0 --port 8000bash

   Expect the log to confirm 'Using FLASHINFER_CUTLASS NvFp4 MoE backend', the draft 'Loading drafter model...' then 'Detected EAGLE model without its own embed_tokens/lm_head. Sharing target model ... weights', and 'GPU KV cache size: 393,261 tokens ... 1.50x'. The cudagraph line 'FULL_AND_PIECEWISE is not supported with spec-decode ... setting cudagraph_mode=PIECEWISE' is expected and harmless.

4. 4

   Wait out the first-boot flashinfer JIT (do not kill it)

   On the first serve of this vLLM/flashinfer version the log freezes for 10-20 minutes after 'torch.compile took NN s', the GPU sits at ~12 W / 0%, and the engine's main thread blocks in anon_pipe_read. This looks exactly like a wedge and it is not: flashinfer 0.6.13 ships no sm_121a cubins and is compiling them from source. It is cached afterwards, so later boots start in ~3 min.

   # proof it's building, not hung:
   pgrep -af 'ninja|cicc|nvcc' | grep -o 'arch=compute_121a,code=sm_121a' | head -1
   du -sh ~/.cache/flashinfer/0.6.13/121a   # grows as it builds
   # vLLM eventually logs e.g. 'Initial profiling/warmup run took 894.32 s'bash

   py-spy needs root on these boxes; to prove liveness without sudo read /proc/<engine-pid>/wchan and its child process list instead.

5. 5

   Smoke test, warm up, then confirm the speculator is accepting

   Confirm coherence, warm the kernels (spec-decode JITs extra Triton kernels on first use), then read the acceptance rate — that number is the whole reason decode is 3x.

   curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
     -d '{"model":"laguna-nvfp4","messages":[{"role":"user","content":"capital of France? one word"}],"max_tokens":8,"temperature":0}'
   # -> "Paris"
   
   # a few hundred tokens of throwaway traffic to warm kernels, then:
   curl -s localhost:8000/metrics | grep -E 'spec_decode_num_(draft_tokens|accepted_tokens)_total'
   # accepted / drafted ~= 0.67 on code at k=6
   
   free -g | sed -n '2p'   # steady-state resident once the API is up (~48 GiB used)bash

   Acceptance is content-dependent: ~67% on code (this recipe's workload), but reasoning/thinking prose drafts far worse. Laguna's thinking is off by default in the chat template, which is the right setting for a coding endpoint. Warm before benchmarking — a cold pass can invert a tuning conclusion.

Key vLLM parameters

API usage

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"laguna-nvfp4",
       "messages":[{"role":"user","content":"Write a Python function that parses a CSV into typed dicts."}],
       "max_tokens":512, "temperature":0}'
# Laguna's chat template defaults enable_thinking=false; pass
# chat_template_kwargs.enable_thinking=true per-request to turn on interleaved reasoning.bash

curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"laguna-nvfp4",
       "messages":[{"role":"user","content":"Complete this module: import csv\n\ndef parse_csv(path):"}],
       "max_tokens":256, "temperature":0, "stream":true,
       "stream_options":{"include_usage":true}}'

# decode_tps = (completion_tokens - 1) / (t_last_token - t_first_token)
curl -s localhost:8000/metrics | grep spec_decode_num_accepted_tokens_total
# measured: 132.2 tok/s at a 2048-token prompt; 67.2% acceptance, 5.03 tokens/stepbash

Troubleshooting

'Model architectures [DFlashLagunaForCausalLM] are not supported.'

You are on vLLM < 0.25.1. The Laguna DFlash draft class (laguna_dflash) first ships in 0.25.1 via upstream PR #46853 (merged 2026-07-03). Upgrade the venv; do not try to back-port onto 0.24.0 — a hand back-port loads and runs but drafts at ~0% acceptance (numerically wrong against newer internals).

Server log freezes for ~15 min after 'torch.compile took NN s'; GPU at ~12 W / 0%; looks hung

It is flashinfer 0.6.13 JIT-compiling sm_121a kernels (no prebuilt cubins for GB10). Confirm with `pgrep -af 'ninja|cicc|nvcc'` showing arch=compute_121a,code=sm_121a, and watch ~/.cache/flashinfer/0.6.13/121a grow. vLLM reports it as part of 'Initial profiling/warmup run took ~894 s'. Cached afterwards — later boots are ~3 min. Do not kill it.

Decode barely improves when you switch NVFP4 -> FP8 or INT4 (all ~42 tok/s no-draft)

Expected, and not a bug. All three poolside quants share an ignore list that leaves attention (q/k/v/o/g_proj) in BF16. On this MoE only 8 of 256 experts fire per token, so the quantized experts are ~16% of per-token reads while the unquantized BF16 attention is ~82% — the bandwidth ceiling is ~78 tok/s regardless of the expert precision. Run `python3 bench/ckpt_bytes.py <checkpoint>` to see the split. The speed lever here is the DFlash draft, not the quantization.

k=1 (or the first k you tried) gives a modest speedup and you ship it

Decode is not monotonic in num_speculative_tokens. On this draft the peak is k=6 (132.2 tok/s), while k=3 is only 98.7 and k=1-scale drafting leaves a third of the gain on the table. Sweep k until decode regresses (k=8 falls back to 112.7 here), then publish one step past the peak. Read per-position acceptance from /metrics: stop raising k once the marginal position nears a coin flip (~0.46 at k=6's position 5).

A very long prompt takes a minute before the first token

That is prefill, and it is honest: decode stays fast but TTFT grows with prompt length — measured 1.5 s at 2K, 3.1 s at 32K, 18.6 s at 128K, 45.1 s at 256K on this config. The context genuinely serves; it just costs compute to ingest. If your workload sends huge prompts, that latency is the thing to budget for, not decode.

SSH to the node times out during banner exchange (ping still works)

Unified-memory swap-wedge: the box overcommitted and userspace (sshd) is starved while the kernel still answers ping. On a single node this usually means an uncapped process ran away — always launch the server (and any local eval that executes generated code) under `systemd-run --scope -p MemoryMax=... -p MemorySwapMax=0`. MemorySwapMax=0 converts the overcommit into a clean OOM-kill instead of a thrash. If it is already wedged, a power-cycle is the reliable recovery.

pkill takes down your own SSH session

`pkill -f 'vllm serve'` over SSH matches the remote command's own argv. Use a character class: `pkill -f '[v]llm serve'`.