Training
Expert parallelism (EP)
Offload / distributeParallelismReferenceDistribute an MoE's experts across devices and dispatch each token to wherever its expert lives — the MoE-native way to scale out.
On the Spark
The natural axis for GLM-5.2's 256 experts; pairs with the 2-bit expert planes and pool-pruning in our 2-Spark run.
- Objective
- Offload / distribute
- Relieves
- CapacityBandwidth
- Format
- MoE experts sharded across devices
- Granularity
- Route tokens to the device holding the chosen expert
- Lifecycle
- PTQ · no gradients
- Calibration
- No calibration
- Compression
- None — splits across N devices
- Quality
- 100% — same weights, sharded and recombined exactly
- Hardware
- All-to-all token routing across the interconnect
- Runtimes
- vLLMSGLang
A worked Spark recipe for this method hasn't been written yet — it lives here as a reference point in the ontology.