Glossary
docs/glossary.mdPart decoder ring, part field guide to the LLM-inference space — what each benchmark number means, the levers (format, method, engine, architecture) that move it, and the acronyms that fly around GPU clusters. Grouped by the problem each idea is trying to solve.
The metrics
What a benchmark row measures.
| Term | Meaning | Bound by |
|---|---|---|
| Prefill (tok/s) | How fast the prompt is ingested before any output | Compute (FLOPs) |
| Decode (tok/s) | Generation speed once tokens are flowing | Memory bandwidth |
| TTFT | Time to first token ≈ promptTokens ÷ prefillTps | Prefill |
| ITL | Gap between streamed tokens = 1000 ÷ decodeTps | Decode |
| Total throughput | Sum of all streams' tok/s under concurrency | Batching efficiency |
On the Spark: decode is your ceiling. With only 273 GB/s of memory bandwidth, generation speed — not compute — is what limits real single-user use. Prefill is comparatively cheap, so at batch = 1 you're almost always watching decode tok/s.
Quantization formats
How shrunk weights are stored — determines bytes/token and which compute path exists.
| Format | What it is | Runs on |
|---|---|---|
| Q4_K_M, Q8_0 | GGUF block formats; llama.cpp's native quants | llama.cpp family |
| GPTQ-Int4 / Int4 | 4-bit integers + group scales, safetensors | vLLM, SGLang |
| MXFP4 | Open-standard 4-bit float, blocks of 32, coarse scales | Both (as plain data) |
| NVFP4 | NVIDIA 4-bit float, blocks of 16, FP8 scales — native Blackwell tensor-core type | vLLM, TRT-LLM |
| FP8 | 8-bit float, native on Hopper/Blackwell | vLLM, TRT-LLM |
| BF16 | Full training precision, no quantization | Everything |
On the Spark: go 4-bit. Because decode speed is set by bytes moved over the bus, a 4-bit model generates roughly 4× faster than BF16 and frees memory for context. NVFP4 is Blackwell's native tensor-core format (the fastest path); MXFP4 is the portable fallback that runs anywhere.
Weight vs activation precision
Model names like NVFP4-W4A16 tell you two precisions — W is the stored weights, A is the activations (the data being multiplied).
| Scheme | Weights | Activations | What it buys |
|---|---|---|---|
| W4A16 | 4-bit | 16-bit (BF16/FP16) | Less memory + less bandwidth; math still runs at 16-bit |
| W4A4 | 4-bit | 4-bit | The above plus faster compute (uses the FP4 tensor cores) |
| W8A8 | 8-bit | 8-bit | Milder compression, high quality; common FP8 serving setup |
On the Spark: W4A16 is the sweet spot. It quarters the bytes moved per token — the thing that actually caps decode — while keeping activations at 16-bit for quality. W4A4 additionally speeds up prefill (4-bit tensor-core matmul), but quantizing activations hurts accuracy and needs calibration, so it only pays off for long-prompt or serving workloads.
Quantization methods
How values are chosen — determines quality, not speed.
| Method | Effort | Who does it |
|---|---|---|
| Plain rounding | Minutes, no data | Anyone (most GGUFs) |
| GPTQ / AWQ / imatrix | Hours + calibration text; compensates rounding error | Anyone (community repos) |
| QAT | Continued training with quantization simulated | Vendors only (e.g. Google's Gemma QAT) |
On the Spark: since you're running 4-bit to stay under the bandwidth ceiling, claw back the lost accuracy for free — a calibrated quant (GPTQ / AWQ / imatrix) or a QAT release costs the same tok/s as plain rounding but generates noticeably better output.
The two stacks
Format determines engine family.
| GGUF world | safetensors world | |
|---|---|---|
| Engines | llama.cpp, ollama, LM Studio | vLLM, SGLang, TensorRT-LLM |
| Wins at | Decode (mature quant kernels) | Prefill/TTFT (vendor GEMM kernels, batching) |
| Audience | Local/enthusiast | Server/production |
On the Spark: for single-user work the GGUF stack (ollama / llama.cpp) is the pragmatic default — simplest setup and strong decode. Reach for vLLM / TensorRT-LLM when you want NVFP4's native Blackwell path or are serving several requests at once.
Architecture & hardware
| Term | Meaning |
|---|---|
| Dense | Every parameter used for every token (27B → reads all 27B) |
| MoE / A3B | Mixture-of-Experts; "A3B" = ~3B active per token — decode speed follows active, not total |
| MTP | Multi-token prediction — extra head enabling speculative decoding (our next big experiment) |
| Unified memory | CPU+GPU share one 128 GB pool @ 273 GB/s — decode ceiling = 273 ÷ active-weight GB |
On the Spark: this lever dominates the others. Decode ceiling = 273 GB/s ÷ active-weight GB, so a dense 27B (reads all 27B every token) crawls while an MoE like A3B moves only ~3B and flies. And the 128 GB unified pool fits models a consumer GPU can't — pick architecture first, everything else second.
Interconnect & networking
How GPUs and nodes talk to each other — the fabric under multi-node training and serving.
| Term | Full name | What it is |
|---|---|---|
| RDMA | Remote Direct Memory Access | One machine reads/writes another's memory directly over the network, bypassing both CPUs and the OS kernel — near-zero-copy, microsecond-latency transfers. The backbone of multi-node GPU training. |
| RoCE | RDMA over Converged Ethernet | RDMA carried over ordinary Ethernet instead of InfiniBand; RoCEv2 is routable over UDP/IP. A cheaper, more familiar fabric for GPU clusters that still want RDMA's low latency — at the cost of needing a well-tuned, lossless network. |
| NCCL | NVIDIA Collective Communications Library | NVIDIA's library for the collective ops multi-GPU training leans on — all-reduce, all-gather, broadcast — tuned to move gradients and activations over NVLink, PCIe, or RDMA fabrics (RoCE/InfiniBand). The layer frameworks call so they don't hand-roll the interconnect. |
On the Spark: these matter the moment you go past a single box. Two Sparks lashed together for tensor/pipeline parallelism lean on exactly this class of interconnect — the link bandwidth between nodes becomes the new ceiling once a model no longer fits in one 128 GB pool.
Model parallelism
How you split a model that's too big for one machine across several.
| Term | Full name | What it is |
|---|---|---|
| PP | Pipeline parallelism | Slices the model by layers, like an assembly line. GLM-5.2 has 78 layers: Spark #1 holds layers 0–39, Spark #2 holds 40–77. A token flows through stage 1's layers, then one intermediate vector (6144 numbers) crosses the 200G link and stage 2 finishes. Cheap handoff — one vector per token. |
| PP2 / PP4 | (stage count) | Just names the number of pipeline stages. PP2 = two Sparks in a chain; upstream's "PP4" meant four GPUs chained. |
| TP | Tensor parallelism | Splits every layer across the GPUs instead — each layer's matrices are sharded and the GPUs swap partial results within every single layer. Much heavier communication, so it needs NVLink-class interconnect; over a two-node link it's far more expensive than PP's one-vector handoff. |
On the Spark: we run PP2 across the two Sparks — the 200G link can't feed TP's per-layer chatter, but PP's one-vector-per-token handoff crosses it cheaply. The cost is the pipeline bubble: with a single conversation only one stage works at a time (stage 2 idles while stage 1 computes, and vice versa), which is why single-stream sits around ~5 tok/s while 8 parallel requests reach ~17 — concurrency fills the bubble. MTP-under-PP (see MTP under Architecture & hardware) is the current debugging saga: making speculative decoding work when the drafter lives on the last stage but the tokens get assembled back on the first.
Attention mechanisms
How a model attends over its context — the main lever on KV-cache size, which sets how much memory a long prompt costs.
| Term | Full name | What it is |
|---|---|---|
| MHA | Multi-Head Attention | The original transformer attention: every query head has its own key and value heads. Highest quality, but the KV cache grows with the head count — the memory hog long contexts fight. |
| GQA | Grouped-Query Attention | Several query heads share one key/value head, shrinking the KV cache vs full MHA while keeping most of the quality. The Llama-family default. |
| MLA | Multi-Head Latent Attention | DeepSeek's scheme: compress K/V down into a small shared latent vector and cache that instead of full K/V. A dramatically smaller KV cache than even GQA, decompressed on the fly during attention. |
| CSA | Compressed Sparse Attention | Attention that both compresses the KV representation and attends to only a sparse subset of past tokens — cutting memory and compute for very long contexts. |
| HCA | Heavily Compressed Attention | Aggressive KV-cache compression aimed at squeezing long contexts into limited memory. (Less standardized than the above — treat the exact definition as paper-specific.) |
On the Spark: with 273 GB/s of bandwidth and a fixed 128 GB pool, the KV cache competes with weights for both space and bus time. Anything in this column that shrinks the cache — GQA, MLA — directly buys you longer context or higher batch at the same decode speed.
Sparsity & pruning
Making a big model cheaper by using — or keeping — only part of it.
| Term | Full name | What it is |
|---|---|---|
| MoE | Mixture of Experts | Only a few of many expert sub-networks fire per token ("A3B" = ~3B active; see Architecture & hardware above). Decode speed follows the active weights, not the total. |
| REAP | Router-weighted Expert Activation Pruning | Prunes whole experts out of an MoE using how often and how heavily the router actually activates each one — shrinking total size while keeping the experts that carry the load. A way to slim an MoE to fit a smaller memory budget. |
On the Spark: pruning like REAP is a direct play for the 128 GB pool — drop the dead-weight experts and a model that didn't fit suddenly does, without touching the active-parameter count that sets decode speed.