* segment sizes marked with an asterisk are estimates pending a measured run
Estimated per-node footprint, weights + KV sharded across the Spark (tensor parallel). Overhead ~8 GB/node; KV at FP16.
one MoE layer · 320 experts, 8+1 fire per token · showing 256
each expert ≈ 16M params
resident: 48 MoE layers × 320 experts × 16M ≈ 241.6B
active/token: 48 × 8 × 16M ≈ 6.0B experts
+ 8.7B always-on dense (attention, embeddings, shared stack) = 14.7B active
Eval scores (compare all)
Reported
Recipes (all recipes)
Worked deployments that run this model on DGX Spark — each shows how it fills each node's unified memory.
Optimize
250B params, ~501 GB at bfloat16 and still ~125 GB at 4-bit — one Spark can't hold it; you need surgery or more Sparks.
Make it fit
Get the weights (and headroom for KV) inside the 128 GB pool — the hard gate.
Prune experts (then quantize)
With 320 routed experts and only 8 firing per token, REAP-style pruning cuts total params without touching active compute.
pruning to half the experts ≈ halves the routed-expert weights, ~69 GB at 4-bit
Split or stream it
Even 4-bit is ~125 GB against a 110 GB usable budget; shard across ≥2 Sparks or stream cold weights from NVMe.
Go below 4 bits
Codebook/outlier methods reach 2–3 bpw when 4-bit still overflows. — for MoE, the proven shape is a 2-bit sign-symmetric codebook on the routed experts with the dense stack kept at FP8 (vLLM-Moet).
~69 GB with 2-bit experts + FP8 dense (est.)
Make it fast
Once it fits: fewer bytes per token and fewer decode steps against 273 GB/s.
Add a speculative decoder
Decode is bandwidth-bound at ~37.1 tok/s ceiling; a trained EAGLE-style draft multiplies tokens per weight-read.
Tame the KV cache
At 1048576 tokens the KV cache alone is ~206 GB at FP16; quantize or evict.
Make it yours
Change what the model does — adapt, edit, or steer it — independent of size.
Fine-tune it
LoRA/QLoRA adapts behavior on one Spark without touching the base weights.
Edit or steer it
One-shot weight math — no gradients, no corpus.