Training
Pipeline parallelism (PP)
Offload / distributeParallelismReferenceGive each device a contiguous block of layers and stream micro-batches through the stages — lighter on interconnect than tensor parallel.
On the Spark
Our earlier 2-Spark config used PP2 via Ray before we switched to TP2 for lower single-request latency.
- Objective
- Offload / distribute
- Relieves
- Capacity
- Format
- Inter-layer stage sharding + micro-batching
- Granularity
- Contiguous layer blocks per device
- Lifecycle
- PTQ · no gradients
- Calibration
- No calibration
- Compression
- None — splits across N devices
- Quality
- 100% — same weights, sharded and recombined exactly
- Hardware
- Tolerates slower links than TP; needs micro-batching to hide bubbles
- Runtimes
- vLLMTensorRT-LLMSGLang
A worked Spark recipe for this method hasn't been written yet — it lives here as a reference point in the ontology.