Training
The resources you can run out of
Required — does it fit?
RequiredCan the weights fit — to load and transform them, and for the finished model to run. The gate on whether you can do anything at all.
Quality of life — how well does it run?
Bytes streamed per token at decode — the Spark's ~273 GB/s LPDDR5x is the usual single-user bottleneck.
FLOPs — dominates prefill and large-batch serving.
Runtime state that grows with context × batch; binds at long context.
Reduce model size
Quantization
GGUF K-quants
llama.cpp mixed per-layer bit allocation — the Ollama / on-device path.
Dynamic quantization (importance-aware)
Spend bits where they hurt — nominal 1-bit quants that are really ~2.3 bpw in the right places.
AWQ
Protects the ~1% salient weights by activation scale; best-in-class INT4.
GPTQ
Hessian-based layerwise error compensation; the classic 4-bit weight quant.
SmoothQuant
Migrates activation outliers into the weights to unlock W8A8.
Rotation / transform quant
Suppress outliers via rotations (QuaRot, SpinQuant, QuIP#, FlatQuant) to enable low-bit activations.
bitsandbytes NF4 / LLM.int8
Zero-calibration load-time quant; NF4 is the base for QLoRA fine-tuning.
AQLM / QuIP# (2-bit codebook)
Codebook / vector quantization pushing weights toward ~2 bits.
SpQR
Variable bitrate for weights: isolate the sensitive outliers in high precision and push everything else to ~3 bits.
SqueezeLLM
Non-uniform quantization that places quant levels by second-order sensitivity, with rare outliers kept sparse in FP16 — more bits, effectively, where they matter.
FP4 microscaling
Pruning / sparsity
REAP expert pruning
One-shot router-weighted pruning that drops an MoE's least-useful experts.
SparseGPT
One-shot layerwise unstructured pruning to 50% with error compensation.
Wanda
Prune by weight × activation-norm — no backprop, no weight update.
N:M (2:4) sparsity
2-of-4 structured sparsity — the one sparsity pattern NVIDIA actually accelerates.
Offload / distribute
Offloading
Tiered expert residency
Treat the GPU as an expert cache, spilling cold MoE experts to host RAM then NVMe (ktransformers, vLLM-Moet).
Partial layer offload
Pin the first N layers on the GPU and run the rest from host RAM (llama.cpp --n-gpu-layers, accelerate device_map).
Parallelism
Tensor parallelism (TP)
Split every layer's matmuls across devices and all-reduce the partials — the workhorse for fitting one model on N GPUs.
Pipeline parallelism (PP)
Give each device a contiguous block of layers and stream micro-batches through the stages — lighter on interconnect than tensor parallel.
Expert parallelism (EP)
Distribute an MoE's experts across devices and dispatch each token to wherever its expert lives — the MoE-native way to scale out.
Reduce KV cache size
KV compression
KV-cache quantization
Quantize the KV cache to fit longer context or a larger batch.
Token eviction (H2O, StreamingLLM)
Drop low-impact tokens from the KV cache to cap long-context memory.
MLA / GQA (architectural KV)
Grouped-query (GQA/MQA) and low-rank latent KV (DeepSeek MLA) baked into the architecture.
Accelerate decoding
Speculative decoding
Draft-model speculative decoding
A small, fast draft model proposes tokens; the big model verifies a whole chunk in one pass — the original speculative decoding.
Medusa heads
Bolt several extra prediction heads onto the model to guess the next few tokens in parallel — no separate draft model.
EAGLE
Draft at the feature (hidden-state) level instead of the token level — higher acceptance than Medusa, the current speculative-decoding SOTA.
Multi-Token Prediction (MTP)
The model is pretrained to emit several tokens at once (DeepSeek-style MTP), so it self-speculates — no draft model or extra head needed.
DSpark
DeepSeek's drafter: a parallel backbone proposes a whole block at once, a lightweight serial head glances one token back to stop the tail drifting, and a confidence head schedules how far to verify — so it doesn't pay to draft-and-discard past the acceptance knee.
Create a net new model
Edit or adapt a model
Fine-tuning
Supervised fine-tuning (SFT)
Instruction / domain tuning: keep training the model on curated input→output pairs to teach a task or style.
Preference tuning (RLHF / DPO)
Optimize the model against preference or reward signals (RLHF, DPO, RLAIF) — how agentic and tool-call behaviors like NSC-ACE get shaped.
Safety / alignment tuning
Train or calibrate the model toward safe/compliant responses on adversarial prompts (e.g. SABER) — the inverse of abliteration.
Weight editing
Abliteration
Discover the 'refusal direction' from paired prompts and orthogonalize the attention output projections against it — uncensoring without retraining.
Knowledge editing (ROME / MEMIT)
Locate where a fact is stored and rewrite it in place with a closed-form weight update, no fine-tuning.
Control vectors / steering
Derive an activation-space direction (honesty, sentiment, refusal) and add it to the residual stream — bakeable into weights or applied live.