howtospark
models / thinkingmachines/Inkling-2-Small
?

594B params~30B activemodeloptMoE 384×8GQA 48:81.0M ctxmultimodalapache-2.0
Sparks
Users
~608 GB
~311 GB
~185 GB
~176 GB
~167 GB
110 usable
REAP dialno pruning · 384 experts
Spec decode
Context33K / 1.0MKV
On the Sparks — 1× · 2-bit experts + NVFP4 dense · 33K ctx · FP16 KV~0 ctx fits usable · ~0 users fit
spark
Expert planes144.1* GB2-bit
110 usable
+57 over
167* / 110 GB
152% of usable
Expert planesDense weightsKV cacheActivations + graphsOS reserve

* segment sizes marked with an asterisk are estimates pending a measured run

Estimated per-node footprint, weights + KV sharded across the Spark (tensor parallel). Overhead ~8 GB/node; KV at FP16.

layer map
48 layersMoE FFN
in
out

one MoE layer · 384 experts, 8 fire per token · showing 256

each expert ≈ 31M params

resident: 48 MoE layers × 384 experts × 31M ≈ 576.2B

active/token: 48 × 8 × 31M ≈ 12.2B experts

+ 17.8B always-on dense (attention, embeddings, shared stack) = 30.1B active

Optimize
Verdict

594B params, ~594 GB at already modelopt-quantized and still ~297 GB at 4-bit — one Spark can't hold it; you need surgery or more Sparks.

Make it fit

Get the weights (and headroom for KV) inside the 128 GB pool — the hard gate.

Prune experts (then quantize)

With 384 routed experts and only 8 firing per token, REAP-style pruning cuts total params without touching active compute (params include vision/audio towers).

pruning to half the experts ≈ halves the routed-expert weights, ~163 GB at 4-bit

Split or stream it

Even 4-bit is ~297 GB against a 110 GB usable budget; shard across ≥3 Sparks or stream cold weights from NVMe.

Go below 4 bits

Codebook/outlier methods reach 2–3 bpw when 4-bit still overflows. — for MoE, the proven shape is a 2-bit sign-symmetric codebook on the routed experts with the dense stack kept at FP8 (vLLM-Moet).

~162 GB with 2-bit experts + FP8 dense (est.)

Make it fast

Once it fits: fewer bytes per token and fewer decode steps against 273 GB/s.

Add a speculative decoder

Decode is bandwidth-bound at ~18.2 tok/s ceiling; a trained EAGLE-style draft multiplies tokens per weight-read.

Make it yours

Change what the model does — adapt, edit, or steer it — independent of size.

Fine-tune it

LoRA/QLoRA adapts behavior on one Spark without touching the base weights.