Training
Tensor parallelism (TP)
Offload / distributeParallelismReferenceSplit every layer's matmuls across devices and all-reduce the partials — the workhorse for fitting one model on N GPUs.
On the Spark
How we put GLM-5.2 (753B) on 2 Sparks: TP2 over the 200G RoCE fabric, ~111 GiB/rank of ~114 usable — fits, but only just.
- Objective
- Offload / distribute
- Relieves
- CapacityCompute
- Format
- Intra-layer weight sharding + all-reduce
- Granularity
- Split each matmul across devices
- Lifecycle
- PTQ · no gradients
- Calibration
- No calibration
- Compression
- None — splits across N devices
- Quality
- 100% — same weights, sharded and recombined exactly
- Hardware
- Fast, low-latency interconnect (NVLink / RoCE / IB) — all-reduce every layer
- Runtimes
- vLLMTensorRT-LLMSGLang
A worked Spark recipe for this method hasn't been written yet — it lives here as a reference point in the ontology.