Training
Knowledge distillation
DistillationReferenceTrain a smaller student on a teacher's outputs; fused with quant as QAD to recover FP4 accuracy.
- Relieves
- CapacityBandwidthCompute
- Targets
- Weights
- Format
- Smaller student model
- Granularity
- Whole-model
- Lifecycle
- QAT / QAD · needs gradients
- Calibration
- Full corpus
- Compression
- Arbitrary (student size)
- Quality
- High with a good teacher; expensive to produce
- Hardware
- Training-grade compute
- Runtimes
- Transformers
A worked Spark recipe for this method hasn't been written yet — it lives here as a reference point in the ontology.