Kimi Linear 48B-A3B — BF16

1 × Spark96K ctx29.4 tok/s@ 2KvLLMthinking offvanilla

Moonshot's Kimi Linear 48B-A3B (49B MoE, ~2.7B active) in BF16 on one DGX Spark, at a 98,304-token context — 29.4 tok/s single-stream decode, with the KV cache costing only 8.2 KB per token.

Kimi Linear is the rare model where the usual Spark tradeoff runs backwards. Normally the weights fit easily and you fight for KV cache; here the weights are 91.53 GiB of a 114 GiB usable pool and the KV cache is almost free. That inversion comes from the architecture: 27 layers, of which 20 are Kimi Delta Attention (a gated linear attention whose state is fixed-size per sequence) and 7 are MLA with a 512-wide compressed latent. A token of context costs 8,064 bytes across the whole model, so a 5 GiB pool holds 656,759 tokens — measured, not modeled. There is no fight for context to be had, which is why this recipe pins the KV pool small and hands the rest back rather than tuning it. What you get for the 92 GB download is a 49B-parameter model that decodes like a 3B one. Only ~2.7B parameters are active per token (top-8 of 256 routed experts plus a shared expert), so the per-token read is about 5.4 GB, which puts a hard bandwidth ceiling near 51 tok/s on GB10's ~273 GB/s. Measured single-stream decode is 29.4 tok/s at a 2,048-token prompt, or 58% of that ceiling, and it barely fades as the prompt grows: 29.5 / 29.4 / 29.2 tok/s at 512 / 2,048 / 8,192 tokens, then 28.5 at 32,768 and 26.3 at 98,304. (The first three come from a serve at --max-model-len 524288 with a 5 GiB KV pin; the last two from the config this page ships. Both are on the same node, same engine, same day.) An 11% decode decay across a 192x increase in prompt length is the linear-attention promise showing up in a measurement. What does grow is TTFT, and steeply: 0.7 s at 2,048 tokens, 8.9 s at 32,768, 37.8 s at 98,304. On this model long context costs you the wait before the first token, not the speed after it. The honest limitation is the one this page is careful about. The vendor advertises a 1,048,576-token window, and the memory for it genuinely exists — at 8,064 bytes/token even a full million tokens needs under 8 GiB of KV — but the prefill path does not survive prompts that long. We bracketed it rather than guessing: with no chunk bound, a 131,072-token prompt killed the engine about 45,000 tokens in, with the KV pool only 6.8% full. Adding --max-num-batched-tokens 4096 moved the ceiling a long way up — a 98,304-token prompt then completed normally — but a 130,000-token prompt still killed it. So 98,304 is what this recipe claims, because 98,304 is what we sent and got an answer to. vLLM 0.26.0 is worse: it will not start this model at all, dying without a traceback immediately after torch.compile whether or not the KV pool is pinned. If you need the full million-token window on a Spark today, this is not yet the configuration that delivers it. One thing this model does not offer is a speculative decoder. config.json sets num_nextn_predict_layers: 0, there is no MTP head in the checkpoint and no published EAGLE/DFlash draft for it, so the single biggest decode lever on the Spark is unavailable — 29.4 tok/s is what this model does. Note also that spec decode on a KDA linear-attention model is not merely absent but architecturally awkward: the recurrent state has to be partitioned into speculative and non-speculative slots, which is engine work rather than a flag.

• 1 x DGX Spark (GB10, sm_121) — no tensor parallelism, no second node
• Weights are a measured 91.53 GiB of the 114 GiB usable pool — the tightest single-Spark fit we serve
• KV costs 8,064 bytes/token across the whole model: 20 of 27 layers are linear-attention and hold no per-token cache
• Measured 131,352 KV tokens per GiB — a 5 GiB pin held a 656,759-token pool
• Measured 29.4 tok/s single-stream decode at a 2,048-token prompt
• Decode barely fades with context: 29.5 / 29.4 / 29.2 / 28.5 / 26.3 tok/s at 512 / 2,048 / 8,192 / 32,768 / 98,304-token prompts
• ~2.7B active parameters of 49B — a ~5.4 GB per-token read, i.e. a ~51 tok/s bandwidth ceiling on GB10
• TTFT is the real long-context cost: 0.7 s at 2,048 tokens, 8.9 s at 32,768, 37.8 s at 98,304
• No speculative decode is possible: num_nextn_predict_layers is 0 and no draft model exists
• The advertised 1,048,576-token window does NOT serve — a 130,000-token prompt crashes the engine mid-prefill
• --max-num-batched-tokens 4096 is load-bearing: it is what got a 98,304-token prompt through at all
• vLLM 0.26.0 cannot start this model — it dies without a traceback right after torch.compile
• Verified 2026-07-27 on vLLM 0.24.0

Software requirements

• 1 x DGX Spark (GB10, sm_121), NVIDIA driver >= 580 (CUDA-13 capable)
• vLLM 0.24.0 with torch 2.11.0+cu130. Do NOT use 0.26.0 — see troubleshooting
• The `hf` CLI to fetch the checkpoint (~92 GB, 20 shards)
• No patches: KimiLinearForCausalLM is natively registered in vLLM, so --trust-remote-code is only needed for the tokenizer and config classes
• Last verified 2026-07-27 on vLLM 0.24.0: weights 91.53 GiB, 2 GiB KV pin -> 253,166-token pool (1.93x at 131,072), decode 29.5 / 29.4 / 29.2 tok/s at 512 / 2,048 / 8,192-token prompts, and a 98,304-token prompt served end to end at 26.3 tok/s

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Kimi-Linear-48B-A3B-Instruct",
       "messages":[{"role":"user","content":"Explain linear attention in two sentences."}],
       "max_tokens":256,"temperature":0}'bash

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Kimi-Linear-48B-A3B-Instruct",
       "messages":[{"role":"user","content":"Write a red-black tree in Python."}],
       "max_tokens":256,"temperature":0,"stream":true,
       "stream_options":{"include_usage":true}}' \
  | grep -o '"completion_tokens":[0-9]*' | tail -1bash

Troubleshooting

Can it really serve the advertised 1,048,576-token context?

vLLM 0.26.0 dies right after 'torch.compile took NN s' with no error

'Free memory on device cuda:0 (95.11/121.69 GiB) on startup is less than desired GPU memory utilization (0.9, 109.52 GiB)'

pkill killed my SSH session instead of the server

Why is decode only 29 tok/s for a model with 2.7B active parameters?

The log says 'Setting attention block size to 1888 tokens'. Is that wrong?

Can I speculative-decode this model?