How much does quantization hurt quality?

<!-- Draft. Numbers live in data/case-studies/quantization-quality.json (runs[]) and drive the charts — keep them out of this prose so the two can't disagree. Delete this note at publish. -->

Background

Every quantization win we've measured so far has been priced in speed and memory: the speed-and-power study showed that dropping precision buys decode throughput and shrinks the footprint, at roughly flat power. But those charts say nothing about whether the model is still right — a faster model that quietly gets more answers wrong is a bad trade, and the whole point of running quantized weights is that the loss is supposed to be small enough not to matter.

This study measures that loss directly, on poolside's Laguna-S-2.1 — a 118B-param agentic coding model. The tension is specific to this hardware: at 118B params, BF16 weights (~236 GiB) don't fit even across both Sparks' 242 GiB of unified memory, so the shipped full-precision artifact we can actually run is poolside's own FP8 build (~118 GiB, two nodes via tp2). The question is how much further we lose going to NVFP4 (~63 GiB) — the only build that fits on a single Spark, and therefore the one most people with one box will actually run.

Method

One variable: weight precision. Everything else held fixed — same base model (poolside/Laguna-S-2.1), same vLLM engine, same eval tasks, same decoding settings (temp 0, pass@1), same extraction/scoring.

Two rungs:

• FP8 — poolside's shipped FP8 checkpoint, served across two Sparks (tp2). This is the full-precision reference: the highest-fidelity Laguna-S build the hardware can hold.
• NVFP4 — 4-bit weights, served on a single Spark.

Fixed eval set, run identically against each served endpoint via EleutherAI's lm-evaluation-harness in API mode (see evals/README.md):

• HumanEval — 164 problems, scored with our humaneval_chat.py last-fenced-block extractor. The stock humaneval_instruct filter mis-extracts Laguna's prose-first output and reports a near-zero pass rate; the chat scorer runs the same official tests on the model's actual code block. This is the same methodology used for the Laguna-XS FP8-vs-NVFP4 comparison, so the numbers are directly comparable.
• GPQA-Diamond — 198 hard-science questions, zero-shot CoT.
• GSM8K — grade-school math, CoT.

Deltas, not absolute scores, are the story — absolute numbers depend on template/extraction details and won't match poolside's vendor tables.

Findings

<prose narrating the charts above once the runs land. Use ### subsections per eval.>

HumanEval

<coding pass@1: FP8 vs NVFP4 delta>

GPQA-Diamond

<reasoning accuracy delta>

GSM8K

<math accuracy delta>

Limitations

• No true BF16 baseline. Laguna-S BF16 doesn't fit on the Sparks, so "full precision" here means poolside's FP8 release, not the raw weights. FP8 is itself a (near-lossless) quantization; any degradation it already carries is invisible to this comparison.
• Confounded topology. FP8 runs across two nodes (tp2); NVFP4 runs on one. Only quality is compared here — not speed — precisely because the serving topologies differ, so a decode-tok/s comparison wouldn't be apples-to-apples.
• Coverage. Three evals sample coding, science reasoning, and arithmetic — not the agentic SWE-bench / terminal-bench tasks Laguna is actually built for (those aren't runnable through our lm-eval harness). Quality loss on long-horizon agentic work could differ from what these short-form evals show.
• NVFP4 here is weight-only (FP4 weights dequantized for the matmul), not w4a4.

Open questions

• Does the quality loss concentrate on the hardest problems, or spread evenly? A per-difficulty breakdown of GPQA/HumanEval would tell us whether NVFP4 is safe for routine work but risky at the frontier.
• How does this compare to the Laguna-XS FP8→NVFP4 delta (91.5→89.0 on HumanEval)? Does the larger S model absorb 4-bit better or worse than XS?
• Would w4a4 NVFP4 (true FP4 tensor cores) — if it could be calibrated to fit — degrade quality more than weight-only NVFP4 does?