howtospark
Training

Tiered expert residency

Offload / distributeOffloadingReference

Treat the GPU as an expert cache, spilling cold MoE experts to host RAM then NVMe (ktransformers, vLLM-Moet).

On the Spark

On the Spark's unified 128 GB there's no VRAM/RAM wall to spill across — this only earns its keep for models that exceed 128 GB, where cold experts page from NVMe.

Objective
Offload / distribute
Relieves
Capacity
Targets
Weights
Format
GPU cache → pinned host RAM → NVMe pack files
Granularity
Per-expert / per-layer paging
Lifecycle
PTQ · no gradients
Calibration
No calibration
Compression
None — relocates, doesn't shrink
Quality
100% — same weights, bit-exact
Hardware
Fast NVMe + PCIe; enough host RAM to pin the hot set
Runtimes
vLLMTransformers
In the wild
vllm-Moet three-tier residency

GPU VRAM expert cache → pinned host RAM 2-bit planes → NVMe pack files; runs GLM-5.2 753B at 28–32 tok/s on 2 GPUs with 140 GiB host RAM.

A worked Spark recipe for this method hasn't been written yet — it lives here as a reference point in the ontology.