Nemotron 3 Nano 30B-A3B — NVFP4

NVIDIA's Nemotron 3 Nano 30B-A3B (31.6B hybrid Mamba-MoE, ~3B active) in NVFP4 on one DGX Spark, serving the full 262,144-token context — 61.2 tok/s single-stream, and still 56.0 tok/s on a 131K-token prompt.

Contributors

@kvncrw@lloyd094

Nemotron 3 Nano 30B-A3B is a hybrid Mamba-Transformer MoE: 52 layers in the pattern `MEMEM*EMEMEM*…`, where 23 are Mamba mixers, 23 are 128-expert MoE blocks (6 experts routed per token plus one shared) and only 6 are full attention. NVIDIA publishes an NVFP4 export of it, and on a DGX Spark that combination lands unusually well — the architecture is bandwidth-friendly on both axes at once. Roughly 3B parameters are active per token, which is what sets decode speed on a bandwidth-bound box; and because only 6 layers keep a KV cache, context costs 3.20 KiB per token instead of the tens of KiB a same-size dense-attention model would want. The practical consequence is that the usual recipe fight does not happen here. There is no climb toward a maximum context and no memory to trade for it: serve at the model's native 262,144-token window, pin a small KV pool, and the node still finishes at 28.3 GiB of 114. Decode barely notices the fill level either — measured warm on one node, single-stream: 61.3 tok/s at a 512-token prompt, 61.2 at 2,048, 60.8 at 8,192, 59.8 at 32,768 and 56.0 at 131,072. That is an 8% decay across a 256x range of prompt length, which is what a mostly-Mamba model buys you and is the reason to pick this checkpoint over a dense one of similar size. What you pay for it is prefill. Time-to-first-token runs 123 ms at 512 tokens, 242 ms at 2,048, 902 ms at 8,192, 3.96 s at 32,768 and 25.9 s at 131,072; a single 260,025-token request completes end to end in 74.8 s. Prefill throughput peaks around 9,100 tok/s at an 8K prompt and falls to ~5,070 tok/s at 128K. So the long window is genuinely usable, but a very long prompt costs most of a minute before the first token appears — budget for it rather than being surprised by it. There is no speculative decoding available: the checkpoint carries no MTP head and NVIDIA ships no draft model for it, so 61 tok/s is the unassisted figure with no lever left unpulled.

• 31.6B hybrid Mamba-MoE, ~3B active per token — 128 routed experts, 6 per token, plus 1 shared
• Only 6 of 52 layers hold a KV cache, so the full 262,144 tokens of context cost 3 GiB
• 61.16 tok/s single-stream decode at a 2,048-token prompt; still 56.0 tok/s at 131,072
• 28.3 GiB total serving working set on a 114 GiB node — weights, KV and overhead together
• vLLM 0.24.0 auto-selects FLASHINFER_CUTLASS for the NVFP4 MoE on GB10; no backend flag needed
• No speculative decoding — no MTP head in the checkpoint and no published draft

Software requirements

• vLLM 0.24.0 — with torch 2.11.0+cu130. CUDA 13 is required for the FP4 MoE kernels — a cu129 build will not serve this checkpoint. Last verified 2026-07-27 on vLLM 0.24.0.
• NVIDIA driver 580.159.03 — GB10 is sm_121; the FLASHINFER_CUTLASS NVFP4 MoE path is auto-selected on this build
• Checkpoint nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 — 5 safetensors shards, ~18 GB. The unsuffixed base repo 401s — the BF16 base is published as -BF16.

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Nemotron-3-Nano-30B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "What is 17*23?"}],
    "max_tokens": 300
  }'

# The model thinks by default. Suppress it per request via the chat template:
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Nemotron-3-Nano-30B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "What is 17*23?"}],
    "chat_template_kwargs": {"enable_thinking": false},
    "max_tokens": 300
  }'

# Take tok/s from usage.completion_tokens over wall time, never by counting
# SSE frames. Warm the server first, then run this a few times.
python3 - <<'PY'
import json, time, urllib.request
body = {
  "model": "Nemotron-3-Nano-30B-A3B-NVFP4",
  "messages": [{"role": "user", "content": "Write 500 words about tidal energy."}],
  "max_tokens": 512, "temperature": 0,
}
req = urllib.request.Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"})
t = time.time()
d = json.load(urllib.request.urlopen(req))
elapsed = time.time() - t
print(f"{d['usage']['completion_tokens'] / elapsed:.2f} tok/s (includes TTFT)")
PY

Troubleshooting

You inherited a serve command with `--max-model-len 10240 --gpu-memory-utilization 0.5` and assume the context is what the hardware allows.

It is not, and on this architecture that is an expensive assumption. That configuration serves 10,240 tokens at 68.3 GiB peak memory. The configuration on this page serves 262,144 tokens — 25.6x the context — at 32.3 GiB peak, and decodes at 61.16 tok/s against 62.13 (both measured on the same node, same engine build, warm, at a matched 2,042-token prompt). Essentially nothing was traded: the KV pool costs 3.20 KiB/token because only 6 of the 52 layers hold one. Before tuning anything on a hybrid model, read `hybrid_override_pattern` in config.json and count the `*` characters — that is the number of full-attention layers, and it is what decides whether context is expensive.

Decode at a 512-token prompt benchmarks slower than at 2,048 tokens.

The run is cold. NVFP4 expert planes fault in from NVMe on first touch, and the first scenario of a benchmark pass pays for all of it. Warm the server with a few hundred tokens of traffic, then run the harness twice and keep the second pass. Warm, the curve is monotonic: 61.26 / 61.16 / 60.81 / 59.79 / 56.00 tok/s at 512 / 2,048 / 8,192 / 32,768 / 131,072-token prompts.

A very long prompt looks like it has hung.

It is prefilling, and on this model prefill is the slow half. Measured TTFT: 123 ms at 512 tokens, 242 ms at 2,048, 902 ms at 8,192, 3.96 s at 32,768, 25.9 s at 131,072 — and a single 260,025-token request took 74.8 s end to end. Prefill throughput peaks near 9,100 tok/s around an 8K prompt and drops to ~5,070 tok/s at 128K. Nothing is wrong; the 262K window is real, it just is not free. `--max-num-batched-tokens` sets the chunk size the prefill is broken into if you need to bound it further.

A request longer than the context window returns HTTP 400.

That is the correct, safe behaviour, and worth knowing you can rely on here — the engine refuses in about half a second rather than dying mid-prefill. The window was bracketed deliberately with real requests: 260,025 tokens serves, and a prompt past 262,144 is rejected. Do not assume the clean 400 is universal across architectures; on some models a long prefill kills the engine instead, so bracket your own.

The model emits its chain of thought inline, in the middle of the answer.

Expected — reasoning is on by default and arrives in DeepSeek-R1-style `<think>` blocks. Two ways out. Per request: pass `"chat_template_kwargs": {"enable_thinking": false}`. Server-side: start with `--reasoning-parser-plugin <checkpoint>/nano_v3_reasoning_parser.py --reasoning-parser nano_v3`, after which the reasoning lands in `choices[0].message.reasoning` and the answer alone in `.content`. Note the field is `reasoning` on vLLM 0.24.0, not `reasoning_content`. The parser is a plugin shipped inside the checkpoint rather than a name built into vLLM, so the `--reasoning-parser-plugin` path is required — `--reasoning-parser nano_v3` alone will not resolve.

You want speculative decoding for a faster decode.

There is nothing to turn on. config.json has no num_nextn_predict_layers and no mtp_num_hidden_layers, the checkpoint has no draft layer beyond its 52, and NVIDIA publishes no EAGLE/DFlash draft for this model. An external draft is also not a flag away on this architecture: spec-decode against a Mamba/linear-attention hybrid requires the recurrent state to be partitioned into speculative and non-speculative slots, which some vLLM linear-attention paths do not populate — where that is missing it surfaces as an assertion on the first generated token rather than as a startup error. Treat 61 tok/s as the unassisted ceiling for this checkpoint.

`pkill -f "vllm serve"` over SSH drops your connection instead of stopping the server.

The remote command's own argv contains the pattern, so pkill matches the shell running it. Use a character class: `pkill -f "[v]llm serve"`. The same trap fires if the *body* of a heredoc you are writing over SSH contains the string `vllm serve` — write the script locally and scp it across instead.