howtospark

Glossary

docs/glossary.md

Part decoder ring, part field guide to the LLM-inference space — what each benchmark number means, the levers (format, method, engine, architecture) that move it, and the acronyms that fly around GPU clusters. Grouped by the problem each idea is trying to solve.

The metrics

What a benchmark row measures.

TermMeaningBound by
Prefill (tok/s)How fast the prompt is ingested before any outputCompute (FLOPs)
Decode (tok/s)Generation speed once tokens are flowingMemory bandwidth
TTFTTime to first token ≈ promptTokens ÷ prefillTpsPrefill
ITLGap between streamed tokens = 1000 ÷ decodeTpsDecode
Total throughputSum of all streams' tok/s under concurrencyBatching efficiency

On the Spark: decode is your ceiling. With only 273 GB/s of memory bandwidth, generation speed — not compute — is what limits real single-user use. Prefill is comparatively cheap, so at batch = 1 you're almost always watching decode tok/s.

Quantization formats

How shrunk weights are stored — determines bytes/token and which compute path exists.

FormatWhat it isRuns on
Q4_K_M, Q8_0GGUF block formats; llama.cpp's native quantsllama.cpp family
GPTQ-Int4 / Int44-bit integers + group scales, safetensorsvLLM, SGLang
MXFP4Open-standard 4-bit float, blocks of 32, coarse scalesBoth (as plain data)
NVFP4NVIDIA 4-bit float, blocks of 16, FP8 scales — native Blackwell tensor-core typevLLM, TRT-LLM
FP88-bit float, native on Hopper/BlackwellvLLM, TRT-LLM
BF16Full training precision, no quantizationEverything

On the Spark: go 4-bit. Because decode speed is set by bytes moved over the bus, a 4-bit model generates roughly 4× faster than BF16 and frees memory for context. NVFP4 is Blackwell's native tensor-core format (the fastest path); MXFP4 is the portable fallback that runs anywhere.

Weight vs activation precision

Model names like NVFP4-W4A16 tell you two precisions — W is the stored weights, A is the activations (the data being multiplied).

SchemeWeightsActivationsWhat it buys
W4A164-bit16-bit (BF16/FP16)Less memory + less bandwidth; math still runs at 16-bit
W4A44-bit4-bitThe above plus faster compute (uses the FP4 tensor cores)
W8A88-bit8-bitMilder compression, high quality; common FP8 serving setup

On the Spark: W4A16 is the sweet spot. It quarters the bytes moved per token — the thing that actually caps decode — while keeping activations at 16-bit for quality. W4A4 additionally speeds up prefill (4-bit tensor-core matmul), but quantizing activations hurts accuracy and needs calibration, so it only pays off for long-prompt or serving workloads.

Quantization methods

How values are chosen — determines quality, not speed.

MethodEffortWho does it
Plain roundingMinutes, no dataAnyone (most GGUFs)
GPTQ / AWQ / imatrixHours + calibration text; compensates rounding errorAnyone (community repos)
QATContinued training with quantization simulatedVendors only (e.g. Google's Gemma QAT)

On the Spark: since you're running 4-bit to stay under the bandwidth ceiling, claw back the lost accuracy for free — a calibrated quant (GPTQ / AWQ / imatrix) or a QAT release costs the same tok/s as plain rounding but generates noticeably better output.

The two stacks

Format determines engine family.

GGUF worldsafetensors world
Enginesllama.cpp, ollama, LM StudiovLLM, SGLang, TensorRT-LLM
Wins atDecode (mature quant kernels)Prefill/TTFT (vendor GEMM kernels, batching)
AudienceLocal/enthusiastServer/production

On the Spark: for single-user work the GGUF stack (ollama / llama.cpp) is the pragmatic default — simplest setup and strong decode. Reach for vLLM / TensorRT-LLM when you want NVFP4's native Blackwell path or are serving several requests at once.

Architecture & hardware

TermMeaning
DenseEvery parameter used for every token (27B → reads all 27B)
MoE / A3BMixture-of-Experts; "A3B" = ~3B active per token — decode speed follows active, not total
MTPMulti-token prediction — extra head enabling speculative decoding (our next big experiment)
Unified memoryCPU+GPU share one 128 GB pool @ 273 GB/s — decode ceiling = 273 ÷ active-weight GB

On the Spark: this lever dominates the others. Decode ceiling = 273 GB/s ÷ active-weight GB, so a dense 27B (reads all 27B every token) crawls while an MoE like A3B moves only ~3B and flies. And the 128 GB unified pool fits models a consumer GPU can't — pick architecture first, everything else second.

Interconnect & networking

How GPUs and nodes talk to each other — the fabric under multi-node training and serving.

TermFull nameWhat it is
RDMARemote Direct Memory AccessOne machine reads/writes another's memory directly over the network, bypassing both CPUs and the OS kernel — near-zero-copy, microsecond-latency transfers. The backbone of multi-node GPU training.
RoCERDMA over Converged EthernetRDMA carried over ordinary Ethernet instead of InfiniBand; RoCEv2 is routable over UDP/IP. A cheaper, more familiar fabric for GPU clusters that still want RDMA's low latency — at the cost of needing a well-tuned, lossless network.
NCCLNVIDIA Collective Communications LibraryNVIDIA's library for the collective ops multi-GPU training leans on — all-reduce, all-gather, broadcast — tuned to move gradients and activations over NVLink, PCIe, or RDMA fabrics (RoCE/InfiniBand). The layer frameworks call so they don't hand-roll the interconnect.

On the Spark: these matter the moment you go past a single box. Two Sparks lashed together for tensor/pipeline parallelism lean on exactly this class of interconnect — the link bandwidth between nodes becomes the new ceiling once a model no longer fits in one 128 GB pool.

Model parallelism

How you split a model that's too big for one machine across several.

TermFull nameWhat it is
PPPipeline parallelismSlices the model by layers, like an assembly line. GLM-5.2 has 78 layers: Spark #1 holds layers 0–39, Spark #2 holds 40–77. A token flows through stage 1's layers, then one intermediate vector (6144 numbers) crosses the 200G link and stage 2 finishes. Cheap handoff — one vector per token.
PP2 / PP4(stage count)Just names the number of pipeline stages. PP2 = two Sparks in a chain; upstream's "PP4" meant four GPUs chained.
TPTensor parallelismSplits every layer across the GPUs instead — each layer's matrices are sharded and the GPUs swap partial results within every single layer. Much heavier communication, so it needs NVLink-class interconnect; over a two-node link it's far more expensive than PP's one-vector handoff.

On the Spark: we run PP2 across the two Sparks — the 200G link can't feed TP's per-layer chatter, but PP's one-vector-per-token handoff crosses it cheaply. The cost is the pipeline bubble: with a single conversation only one stage works at a time (stage 2 idles while stage 1 computes, and vice versa), which is why single-stream sits around ~5 tok/s while 8 parallel requests reach ~17 — concurrency fills the bubble. MTP-under-PP (see MTP under Architecture & hardware) is the current debugging saga: making speculative decoding work when the drafter lives on the last stage but the tokens get assembled back on the first.

Attention mechanisms

How a model attends over its context — the main lever on KV-cache size, which sets how much memory a long prompt costs.

TermFull nameWhat it is
MHAMulti-Head AttentionThe original transformer attention: every query head has its own key and value heads. Highest quality, but the KV cache grows with the head count — the memory hog long contexts fight.
GQAGrouped-Query AttentionSeveral query heads share one key/value head, shrinking the KV cache vs full MHA while keeping most of the quality. The Llama-family default.
MLAMulti-Head Latent AttentionDeepSeek's scheme: compress K/V down into a small shared latent vector and cache that instead of full K/V. A dramatically smaller KV cache than even GQA, decompressed on the fly during attention.
CSACompressed Sparse AttentionAttention that both compresses the KV representation and attends to only a sparse subset of past tokens — cutting memory and compute for very long contexts.
HCAHeavily Compressed AttentionAggressive KV-cache compression aimed at squeezing long contexts into limited memory. (Less standardized than the above — treat the exact definition as paper-specific.)

On the Spark: with 273 GB/s of bandwidth and a fixed 128 GB pool, the KV cache competes with weights for both space and bus time. Anything in this column that shrinks the cache — GQA, MLA — directly buys you longer context or higher batch at the same decode speed.

Sparsity & pruning

Making a big model cheaper by using — or keeping — only part of it.

TermFull nameWhat it is
MoEMixture of ExpertsOnly a few of many expert sub-networks fire per token ("A3B" = ~3B active; see Architecture & hardware above). Decode speed follows the active weights, not the total.
REAPRouter-weighted Expert Activation PruningPrunes whole experts out of an MoE using how often and how heavily the router actually activates each one — shrinking total size while keeping the experts that carry the load. A way to slim an MoE to fit a smaller memory budget.

On the Spark: pruning like REAP is a direct play for the 128 GB pool — drop the dead-weight experts and a model that didn't fit suddenly does, without touching the active-parameter count that sets decode speed.