Laguna-XS 2.1 Q4_K_M (llama.cpp)

Serve poolside's agentic-coding Laguna-XS 2.1 (33B MoE, ~3B active) on one DGX Spark as a 20.3 GB Q4_K_M GGUF: 90.4 tok/s single-stream with no draft, 122.7 with the DFlash speculator, and the model's full native 262,144-token window open the whole time — which, measured, costs about 1 GiB and zero decode speed. Needs a llama.cpp build carrying the laguna architecture port; stock llama.cpp cannot load this model at all.

Same quantized model

• Laguna-XS 2.1 (poolside) — Q8_0
• Laguna-XS 2.1 (poolside) — BF16 (vLLM)
• Laguna-XS 2.1 (poolside) — BF16 (llama.cpp)

Laguna-XS 2.1 is poolside's 33B agentic-coding MoE — 256 routed experts plus one shared, 8 firing per token, so about 3B active — with a native 262K context via YaRN. This recipe serves it through llama.cpp as a self-quantized 20.3 GB Q4_K_M GGUF, and it exists as a deliberate counterweight to the NVFP4 vLLM recipe for the same model, because the two engines answer the speed question in completely different places. Start with the thing nobody publishes: on this checkpoint the *quantization scope*, not the engine, is what sets the no-draft ceiling. Every poolside quant (-FP8, -NVFP4, -INT4) carries an ignore list that leaves q/k/v/o/g_proj in BF16, and since only 8 of 256 experts fire per token, that unquantized attention is ~82% of the bytes a token actually reads — all three decode within 2 tok/s of each other at ~42 tok/s. llama.cpp's Q4_K_M has no such scruples: it quantizes attention too, at ~4.5 bits, and measures 90.4 tok/s here with no draft at all. That is a 2.1x no-draft advantage that comes purely from what got quantized. Then the picture inverts. Both stacks can drive the checkpoint's companion DFlash speculator, but they are not equally good at it: vLLM 0.25.1's native draft class reaches ~67% acceptance, while our llama.cpp port of the same draft reaches 28.8–33.5% on the same code workload — re-measured in this session, so the gap is stable, and it is a deficiency in the port rather than a property of the checkpoint. The result is that llama.cpp gets 1.34–1.36x from the draft (90.4 → 122.7 tok/s at a 512-token prompt) where vLLM gets 3.17x, and vLLM's 133.6 tok/s ends up ahead of llama.cpp's 122.7 despite starting from less than half the baseline. So: llama.cpp if you want the best answer without a speculator, a smaller file, and a three-second warm start; vLLM if you are willing to pin a specific version and want the top number. The second thing this recipe settles is context, and the answer is that there is nothing to settle. The July 2026 configs for this model served at -c 10240, on the assumption that a bigger window would cost memory or speed. Measured here: with -c 262144 the server holds 35 GiB steady state and decodes 90.4 / 86.7 / 77.0 tok/s at 512 / 2,048 / 8,192-token prompts, inside run-to-run noise of the 89.4 / 86.3 / 77.1 measured at -c 10240 on 2026-07-12. Open the whole window. The real long-context cost is prefill, not memory: a 131,104-token prompt takes 178 s to first token (690 tok/s prefill) against 2,405 tok/s at 8K. That prompt and a 250,033-token one both served correctly; 270,043 tokens is rejected with a clean HTTP 400, so the published 262,144 is a bracketed number rather than a config value. The load-bearing caveat is that none of this runs on stock llama.cpp — the laguna architecture is a local port (upstreamed as ggml-org/llama.cpp#25595, not merged at time of writing), and the DFlash draft support sits on top of it.

• 1 x DGX Spark (GB10, sm_121) — 33B MoE in a 20.3 GB Q4_K_M GGUF, ~35 GiB resident
• 90.4 tok/s single-stream with no draft — 2.1x the same model's vLLM NVFP4 build (41.6), because Q4_K_M quantizes attention and the poolside quants do not
• 122.7 tok/s with the DFlash speculator at a 512-token code prompt (1.36x); 118.2 at 2K, 102.9 at 8K
• Draft acceptance 33.5% / 32.7% / 28.8% at 512 / 2K / 8K — well below the ~67% vLLM gets from the same draft; the gap is in our llama.cpp port
• Full native 262,144-token context open costs ~1 GiB and no measurable decode speed vs the old -c 10240 configs
• Context bracketed with real requests: 250,033 tokens serves, 270,043 returns a clean 400
• Long context is prefill-bound, not memory-bound: 178 s to first token at 131K tokens
• Cold start ~15 s, warm restart ~3 s — no JIT wall, unlike the vLLM path's ~15 min first boot
• 50–63 W measured during the runs; an 8K-prompt request costs on the order of 200 J
• Requires a llama.cpp build with the laguna arch port — stock llama.cpp cannot load this model

Software requirements

• 1 x DGX Spark (GB10, sm_121), NVIDIA driver >= 580, CUDA llama.cpp build for sm_121
• llama.cpp with the `laguna` architecture port — verified on build b9891-11-g7ad9bd2 (the server reports b25-7ad9bd2), worktree ~/Dev/llama.cpp-laguna, branch laguna-support. Upstream PR ggml-org/llama.cpp#25595 carries the functional arch code; the DFlash draft support (--spec-type draft-dflash) is local on top of it and is not upstream
• Stock llama.cpp fails to load this GGUF outright — the laguna arch is unknown to it. Serve from the laguna worktree build, NOT ~/Dev/llama.cpp
• ~21 GB free disk for the quantized GGUF + draft; ~88 GB transiently if you convert and quantize from the BF16 checkpoint yourself
• Base model: poolside/Laguna-XS-2.1 (license OpenMDW-1.1). Draft: poolside/Laguna-XS-2.1-DFlash
• No published Q4_K_M GGUF exists for this model — you quantize it yourself (step 2)

Key vLLM parameters

API usage

curl -s localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"laguna",
       "messages":[{"role":"user","content":"Write a Python function that parses a CSV into typed dicts."}],
       "max_tokens":512, "temperature":0}'bash

# tokenize a long document, then send it
curl -s localhost:8080/tokenize -H 'Content-Type: application/json' \
  -d '{"content":"<your long document>"}' \
  | python3 -c 'import json,sys;print(len(json.load(sys.stdin)["tokens"]))'

# past the window you get a clean, specific error rather than a hang:
# {"error":{"code":400,"message":"request (270043 tokens) exceeds the available
#  context size (262144 tokens), try increasing it",
#  "type":"exceed_context_size_error","n_prompt_tokens":270043,"n_ctx":262144}}bash

grep 'draft acceptance' /path/to/server.log | tail -5
# slot print_timing: id  0 | task 14 | draft acceptance = 0.33974
#   (212 accepted / 624 generated), mean len = 5.93bash

Troubleshooting

llama-server refuses the GGUF with an unknown-architecture error

You are on stock llama.cpp. The `laguna` arch is a port that lives in the fork worktree (~/Dev/llama.cpp-laguna, branch laguna-support; upstreamed as ggml-org/llama.cpp#25595, not merged at time of writing). Serve from that build's binary, not ~/Dev/llama.cpp/build/bin/llama-server. This is the most common failure for this recipe and it is easy to hit by muscle memory.

'this custom template is not supported'

Add --jinja. Laguna's chat template is not expressible in llama.cpp's built-in template engine. This affects llama-server and llama-completion equally.

The model answers with an empty string, or a benchmark reports far fewer tokens than max_tokens

Thinking is on. Laguna does interleaved reasoning and llama.cpp puts it in `reasoning_content`, not `content`, so a client reading only `content` sees nothing while the output budget is consumed. Pass --reasoning-budget 0 for coding/agent endpoints. Every number on this page was measured with thinking off; rows measured with it on are not comparable.

You set -c 262144 but requests fail past ~65K tokens

You also set --parallel 4 (or left it above 1). In llama.cpp, -c is the TOTAL context divided across slots, so four slots at -c 262144 gives each request 65,536 tokens. Use --parallel 1 for single-user serving, and check the log line `n_ctx_slot = 262144` rather than trusting the -c you passed.

Draft acceptance is near zero at long prompts but fine at short ones

You are using a draft GGUF converted without the drafter's trained sliding window. Measured: the -noswa conversion drops to 0.001 acceptance at a 7.8K-token prompt while the SWA-512 one holds 0.280. Keep the trained SWA. The merged vLLM reference implementation strips it, which is likely an unreported upstream bug — do not copy that choice into a llama.cpp conversion.

The DFlash speedup is only ~1.35x, but the vLLM recipe for this model claims 3.17x

Both numbers are real and both were measured on this hardware. The difference is acceptance: vLLM 0.25.1's native laguna_dflash reaches ~0.67 on code, this llama.cpp port reaches 0.288–0.335 (re-confirmed 2026-07-27). The port is numerically deficient somewhere in the draft path; the checkpoint is fine. The compensating fact is that llama.cpp's no-draft baseline is already 2.1x vLLM's (90.4 vs 41.6 tok/s), because Q4_K_M quantizes attention while the poolside NVFP4/FP8/INT4 checkpoints leave it BF16. Net: vLLM+DFlash at 133.6 still beats llama.cpp+DFlash at 122.7, but llama.cpp wins decisively if you are not running a speculator.

A long prompt seems to hang for minutes with no output

Expected — this config is prefill-bound at length, not memory-bound. Measured: a 131,104-token prompt takes 178 s to first token at 690 tok/s prefill; a 250,033-token one took 197 s at a reported 607 tok/s, but that request shared a long prefix with the previous one and so benefited from llama.cpp's slot cache — treat the 250K figure as a floor on the true cost, not a clean measurement. At 8K the prefill rate is 2,405 tok/s. A quarter-million-token prompt is a fine trade for a document you ask several questions about and a bad one for interactive work.

You switch to Q8_0 or BF16 for quality headroom and decode collapses

Working as intended on a bandwidth-bound part: decode tracks file size almost exactly on this ~3B-active MoE. Measured at a 512-token prompt: Q4_K_M 90.4 tok/s (20.3 GB), Q8_0 62.9 (35.6 GB), BF16 33.5 (66.9 GB). If you need the bigger quant, note the DFlash draft pays proportionally MORE there (1.71x on BF16 vs 1.36x on Q4_K_M), because the draft overhead is a smaller fraction of the target passes it saves.

pkill kills your SSH session instead of the server

Over SSH your own command line contains the pattern, so `pkill -f llama-server` matches the shell running it. Put the kill in a script file on the remote host. Note that the usual '[l]lama-server' character-class guard does NOT save you if the same command line also contains an unguarded copy of the path — e.g. the serve command you are about to launch in the same one-liner.