Training
KV-cache quantization
KV compressionReferenceQuantize the KV cache to fit longer context or a larger batch.
- Relieves
- KV cache
- Targets
- KV cache
- Format
- KV8 / KV4 (INT8 / INT4)
- Granularity
- Per-token / per-head
- Lifecycle
- PTQ · no gradients
- Calibration
- No calibration
- Compression
- 2–4× KV
- Quality
- KV8 near-lossless; KV4 small loss
- Hardware
- Runtime KV-quant support
- Runtimes
- vLLMTensorRT-LLMSGLang
A worked Spark recipe for this method hasn't been written yet — it lives here as a reference point in the ontology.