Training
GGUF K-quants
QuantizationPlannedllama.cpp mixed per-layer bit allocation — the Ollama / on-device path.
On the Spark
The single-Spark serving route when vLLM lacks a kernel for a brand-new arch.
- Relieves
- CapacityBandwidth
- Targets
- Weights
- Format
- Mixed 2–8 bit (Q4_K_M, Q5_K_M, …)
- Granularity
- Per-block, mixed per-layer bit allocation
- Lifecycle
- PTQ · no gradients
- Calibration
- No calibration
- Compression
- 2–6×
- Quality
- Tunable; Q4_K_M ≈ near-FP16
- Hardware
- None — CPU/GPU split, runs on any silicon
- Runtimes
- llama.cpp
A worked Spark recipe for this method hasn't been written yet — it lives here as a reference point in the ontology. It's on the list.