Concepts
docs/concepts.mdA running glossary of the ideas behind the benchmarks — what each number means, and the levers (format, method, engine, architecture) that move it.
The metrics
What a benchmark row measures.
| Term | Meaning | Bound by |
|---|---|---|
| Prefill (tok/s) | How fast the prompt is ingested before any output | Compute (FLOPs) |
| Decode (tok/s) | Generation speed once tokens are flowing | Memory bandwidth |
| TTFT | Time to first token ≈ promptTokens ÷ prefillTps | Prefill |
| ITL | Gap between streamed tokens = 1000 ÷ decodeTps | Decode |
| Total throughput | Sum of all streams' tok/s under concurrency | Batching efficiency |
On the Spark: decode is your ceiling. With only 273 GB/s of memory bandwidth, generation speed — not compute — is what limits real single-user use. Prefill is comparatively cheap, so at batch = 1 you're almost always watching decode tok/s.
Quantization formats
How shrunk weights are stored — determines bytes/token and which compute path exists.
| Format | What it is | Runs on |
|---|---|---|
| Q4_K_M, Q8_0 | GGUF block formats; llama.cpp's native quants | llama.cpp family |
| GPTQ-Int4 / Int4 | 4-bit integers + group scales, safetensors | vLLM, SGLang |
| MXFP4 | Open-standard 4-bit float, blocks of 32, coarse scales | Both (as plain data) |
| NVFP4 | NVIDIA 4-bit float, blocks of 16, FP8 scales — native Blackwell tensor-core type | vLLM, TRT-LLM |
| FP8 | 8-bit float, native on Hopper/Blackwell | vLLM, TRT-LLM |
| BF16 | Full training precision, no quantization | Everything |
On the Spark: go 4-bit. Because decode speed is set by bytes moved over the bus, a 4-bit model generates roughly 4× faster than BF16 and frees memory for context. NVFP4 is Blackwell's native tensor-core format (the fastest path); MXFP4 is the portable fallback that runs anywhere.
Weight vs activation precision
Model names like NVFP4-W4A16 tell you two precisions — W is the stored weights, A is the activations (the data being multiplied).
| Scheme | Weights | Activations | What it buys |
|---|---|---|---|
| W4A16 | 4-bit | 16-bit (BF16/FP16) | Less memory + less bandwidth; math still runs at 16-bit |
| W4A4 | 4-bit | 4-bit | The above plus faster compute (uses the FP4 tensor cores) |
| W8A8 | 8-bit | 8-bit | Milder compression, high quality; common FP8 serving setup |
On the Spark: W4A16 is the sweet spot. It quarters the bytes moved per token — the thing that actually caps decode — while keeping activations at 16-bit for quality. W4A4 additionally speeds up prefill (4-bit tensor-core matmul), but quantizing activations hurts accuracy and needs calibration, so it only pays off for long-prompt or serving workloads.
Quantization methods
How values are chosen — determines quality, not speed.
| Method | Effort | Who does it |
|---|---|---|
| Plain rounding | Minutes, no data | Anyone (most GGUFs) |
| GPTQ / AWQ / imatrix | Hours + calibration text; compensates rounding error | Anyone (community repos) |
| QAT | Continued training with quantization simulated | Vendors only (e.g. Google's Gemma QAT) |
On the Spark: since you're running 4-bit to stay under the bandwidth ceiling, claw back the lost accuracy for free — a calibrated quant (GPTQ / AWQ / imatrix) or a QAT release costs the same tok/s as plain rounding but generates noticeably better output.
The two stacks
Format determines engine family.
| GGUF world | safetensors world | |
|---|---|---|
| Engines | llama.cpp, ollama, LM Studio | vLLM, SGLang, TensorRT-LLM |
| Wins at | Decode (mature quant kernels) | Prefill/TTFT (vendor GEMM kernels, batching) |
| Audience | Local/enthusiast | Server/production |
On the Spark: for single-user work the GGUF stack (ollama / llama.cpp) is the pragmatic default — simplest setup and strong decode. Reach for vLLM / TensorRT-LLM when you want NVFP4's native Blackwell path or are serving several requests at once.
Architecture & hardware
| Term | Meaning |
|---|---|
| Dense | Every parameter used for every token (27B → reads all 27B) |
| MoE / A3B | Mixture-of-Experts; "A3B" = ~3B active per token — decode speed follows active, not total |
| MTP | Multi-token prediction — extra head enabling speculative decoding (our next big experiment) |
| Unified memory | CPU+GPU share one 128 GB pool @ 273 GB/s — decode ceiling = 273 ÷ active-weight GB |
On the Spark: this lever dominates the others. Decode ceiling = 273 GB/s ÷ active-weight GB, so a dense 27B (reads all 27B every token) crawls while an MoE like A3B moves only ~3B and flies. And the 128 GB unified pool fits models a consumer GPU can't — pick architecture first, everything else second.