Training
Tiered expert residency
Offload / distributeOffloadingReferenceTreat the GPU as an expert cache, spilling cold MoE experts to host RAM then NVMe (ktransformers, vLLM-Moet).
On the Spark
On the Spark's unified 128 GB there's no VRAM/RAM wall to spill across — this only earns its keep for models that exceed 128 GB, where cold experts page from NVMe.
- Objective
- Offload / distribute
- Relieves
- Capacity
- Targets
- Weights
- Format
- GPU cache → pinned host RAM → NVMe pack files
- Granularity
- Per-expert / per-layer paging
- Lifecycle
- PTQ · no gradients
- Calibration
- No calibration
- Compression
- None — relocates, doesn't shrink
- Quality
- 100% — same weights, bit-exact
- Hardware
- Fast NVMe + PCIe; enough host RAM to pin the hot set
- Runtimes
- vLLMTransformers
In the wild
vllm-Moet three-tier residency
GPU VRAM expert cache → pinned host RAM 2-bit planes → NVMe pack files; runs GLM-5.2 753B at 28–32 tok/s on 2 GPUs with 140 GiB host RAM.
A worked Spark recipe for this method hasn't been written yet — it lives here as a reference point in the ontology.