Training
The resources you can run out of
Required — does it fit?
RequiredCan the weights fit — to load and transform them, and for the finished model to run. The gate on whether you can do anything at all.
Quality of life — how well does it run?
Bytes streamed per token at decode — the Spark's ~273 GB/s LPDDR5x is the usual single-user bottleneck.
FLOPs — dominates prefill and large-batch serving.
Runtime state that grows with context × batch; binds at long context.
Quantization
Fewer bits per number — weight-only, weight+activation, or KV.
GGUF K-quants
llama.cpp mixed per-layer bit allocation — the Ollama / on-device path.
AWQ
Protects the ~1% salient weights by activation scale; best-in-class INT4.
GPTQ
Hessian-based layerwise error compensation; the classic 4-bit weight quant.
SmoothQuant
Migrates activation outliers into the weights to unlock W8A8.
Rotation / transform quant
Suppress outliers via rotations (QuaRot, SpinQuant, QuIP#, FlatQuant) to enable low-bit activations.
bitsandbytes NF4 / LLM.int8
Zero-calibration load-time quant; NF4 is the base for QLoRA fine-tuning.
AQLM / QuIP# (2-bit codebook)
Codebook / vector quantization pushing weights toward ~2 bits.
FP4 microscaling
Hardware-coupled 4-bit block formats (MXFP4, NVFP4) for Blackwell.
Pruning / sparsity
Remove weights — unstructured, semi-structured (N:M), or structured.
REAP expert pruning
One-shot router-weighted pruning that drops an MoE's least-useful experts.
SparseGPT
One-shot layerwise unstructured pruning to 50% with error compensation.
Wanda
Prune by weight × activation-norm — no backprop, no weight update.
N:M (2:4) sparsity
2-of-4 structured sparsity — the one sparsity pattern NVIDIA actually accelerates.
Distillation
Train a smaller student to mimic a teacher.
Low-rank
SVD / adapters; often a quant-error compensation term.
KV compression
Compress runtime KV state, not the weights.
KV-cache quantization
Quantize the KV cache to fit longer context or a larger batch.
Token eviction (H2O, StreamingLLM)
Drop low-impact tokens from the KV cache to cap long-context memory.
MLA / GQA (architectural KV)
Grouped-query (GQA/MQA) and low-rank latent KV (DeepSeek MLA) baked into the architecture.