Nemotron 3 Nano 30B-A3B — BF16

The unquantized BF16 build of NVIDIA's Nemotron 3 Nano 30B-A3B on one DGX Spark. It serves the same full 262,144-token context as the NVFP4 recipe — but it costs 68 GiB instead of 28.3, and decodes at 28.9 tok/s instead of 61.2. This is the page that tells you what NVFP4 is actually buying.

This is the reference build of Nemotron 3 Nano 30B-A3B — NVIDIA's own BF16 weights, no quantization step. It exists on this site mostly so the NVFP4 recipe has something honest to be compared against, and the comparison was run on one node, one engine build, both warm, in a single session. The headline is that **you keep the context and you pay for the speed**. The architecture is what makes the long window cheap — 52 layers in the pattern `MEMEM*EMEMEM*…`, of which 23 are Mamba mixers, 23 are 128-expert MoE blocks and only 6 are full attention — and that is a property of the model, not of the numeric format. So BF16 serves the same native 262,144-token window on one Spark with the same 3 GiB pinned KV pool, and a 260,395-token request completes end to end in 88.3 s. What changes is everything on the bandwidth axis: 58.82 GiB of weights resident instead of 18.66, a 68 GiB working set instead of 28.3, and decode at 28.93 tok/s instead of 61.16 at a matched 2,041-token prompt. That is a **2.11x decode penalty** for the unquantized weights. It is worth noting that 2.11x is *less* than the checkpoint-size ratio of 3.15x, and the gap is real rather than measurement slop. Decode time on this model is not purely weight-streaming: the Mamba state updates, the six attention layers, sampling and kernel-launch overhead all cost the same in either build, so shrinking the expert planes by roughly 3.5x moves the total by only 2.11x. If you are budgeting a quantization win on a hybrid Mamba-MoE, expect the byte ratio to overpromise. Decode is nearly flat with fill level, same as the NVFP4 build: 28.97 / 28.93 / 28.80 / 28.37 / 26.55 tok/s at 503 / 2,041 / 8,203 / 32,774 / 131,267-token prompts — 8% decay across a 261x range of prompt length. Prefill is where the long window is paid for: TTFT runs 293 ms at 512 tokens, 456 ms at 2,048, 1.49 s at 8,192, 6.03 s at 32,768 and 32.8 s at 131,072. **Pick this build only if you need it.** The realistic reasons are that you are validating a quantization and need a reference, or you have measured a quality regression on NVFP4 that matters for your workload. Nobody has yet run an eval comparing the two on this model, so "NVFP4 is fine" is currently an assumption on this site, not a measurement. Absent that, the NVFP4 recipe is the better default on a Spark by a wide margin: same context, 2.11x the decode speed, and 40 GiB of the node handed back.

• Same 262,144-token context as the NVFP4 build — the long window comes from the architecture, not the quantization
• 28.93 tok/s single-stream decode at a 2,041-token prompt — 2.11x slower than the NVFP4 sibling's 61.16 on the same node and engine build
• 68 GiB serving working set vs 28.3 GiB for NVFP4; 58.82 GiB of that is weights
• KV costs 6,390 bytes/token here vs 3,281 on NVFP4: that checkpoint declares kv_cache_quant_algo FP8, this one declares nothing
• The 3.15x checkpoint-size ratio only buys 2.11x decode — the non-weight half of a hybrid's decode step does not shrink
• Weight load takes 397 s: at 58.82 GiB the checkpoint exceeds 90% of vLLM's view of available RAM, which disables its prefetch path
• No speculative decoding — no MTP head in the checkpoint and no published draft

Software requirements

• vLLM 0.24.0 — with torch 2.11.0+cu130. Matched deliberately to the NVFP4 recipe's build so the two sets of numbers are comparable. Last verified 2026-07-27 on vLLM 0.24.0.
• NVIDIA driver 580.159.03 — GB10 is sm_121. Nothing FP4-specific is in play on this build — it is a plain BF16 MoE forward pass.
• Checkpoint nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 — 13 safetensors shards, ~59 GB. The unsuffixed base repo 401s — the BF16 base is published as -BF16.

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Nemotron-3-Nano-30B-A3B-BF16",
    "messages": [{"role": "user", "content": "What is 17*23?"}],
    "max_tokens": 300
  }'bash

# The model thinks by default. Suppress it per request via the chat template:
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Nemotron-3-Nano-30B-A3B-BF16",
    "messages": [{"role": "user", "content": "What is 17*23?"}],
    "chat_template_kwargs": {"enable_thinking": false},
    "max_tokens": 300
  }'bash

# The one line that distinguishes this build from the NVFP4 one at runtime.
# Divide the pinned pool by the reported token count:
grep 'GPU KV cache size' /tmp/bf16-serve.log
#   GPU KV cache size: 504,123 tokens
#   3221225472 / 504123 = 6,390 bytes/token   (BF16 KV)
#
# The NVFP4 sibling, same pin, same 6 attention layers:
#   GPU KV cache size: 982,061 tokens
#   3221225472 / 982061 = 3,281 bytes/token   (FP8 KV, from the checkpoint's
#                                              hf_quant_config kv_cache_quant_algo)bash

Troubleshooting

You want to know what NVFP4 actually costs you in quality, and whether this build is worth its 2.11x decode penalty.

Nobody has measured it yet, on this site or anywhere we could find, and we are not going to pretend otherwise. What is measured is the price: 28.93 tok/s against 61.16 at a matched 2,041-token prompt, and 68 GiB of working set against 28.3 — same node, same vLLM 0.24.0, both warm, both measured on 2026-07-27. Running `evals/run_eval.py` against the two endpoints in one session is what would settle it. Until someone does, treat this recipe as the reference build you validate against, not as the one you serve.

Startup takes over seven minutes and the log says auto-prefetch is disabled.

Expected, and it is a unified-memory artifact rather than a fault. vLLM prints `Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS and the checkpoint size (58.82 GiB) exceeds 90% of available RAM (54.33 GiB)`. On a Spark, weights and system memory come out of the same 128 GB pool, so a 59 GB checkpoint is genuinely close to all the RAM vLLM believes it has, and the prefetch path switches itself off. Weight load then takes 397 s. Budget seven and a half minutes for every restart and get the serve command right the first time — the NVFP4 build boots in about three.

A 3 GiB KV pin gives you half the tokens you expected from reading the NVFP4 recipe.

It does, and the geometry is not the reason — both builds have the same 6 full-attention layers with 2 KV heads at head_dim 128. The NVFP4 checkpoint ships an `hf_quant_config` declaring `kv_cache_quant_algo: FP8`, which vLLM honours (`kv_cache_dtype=fp8_e4m3` in the startup config); the BF16 repo ships no such file and gets a BF16 cache. So 504,123 tokens here vs 982,061 there, 6,390 bytes/token vs 3,281. The general lesson: when you compare quant siblings, do not assume KV bytes/token is held constant — quantizing the weights may have quantized the KV too. You can ask for the cheaper cache explicitly with `--kv-cache-dtype fp8` if you want it without the quantized weights.

The benchmark rows report `peakMemoryGb: 15.7`, which is less than the weights.

That figure is wrong for this model and should be ignored — the weight residency alone is 58.82 GiB, so 15.7 GB cannot be a working set. Use `free -g` on the node while serving and subtract the idle baseline instead; that is where the 68 GiB in the memory budget above comes from (72 GiB serving, 4 GiB idle measured on the same node after teardown). Peak-memory reporting is not reliable on GB10 unified memory, where `nvidia-smi` returns `[N/A]` for `memory.used` entirely.

Decode at a 512-token prompt benchmarks slower than at 2,048 tokens.

The run is cold, and at 59 GB of weights there is more to fault in from NVMe than on any quantized build. Warm the server with a few hundred tokens of traffic, then run the harness twice and keep the second pass. Warm, this model is exceptionally reproducible: two consecutive full passes agreed within 0.5% at every prompt length, so if your numbers move more than that, something else is running on the node.

A very long prompt looks like it has hung.

It is prefilling, and on this model prefill is the slow half. Measured TTFT: 293 ms at 512 tokens, 456 ms at 2,048, 1.49 s at 8,192, 6.03 s at 32,768, 32.8 s at 131,072 — and a single 260,395-token request took 88.3 s end to end. Prefill throughput peaks near 5,500 tok/s around an 8K prompt and falls to ~4,000 tok/s at 128K. Nothing is wrong; the 262K window is real, it just is not free.

You batch four streams and the aggregate throughput disappoints.

At 4 concurrent 512-token streams this build gives 17.16 tok/s per stream and 66.2 tok/s aggregate, against 28.97 tok/s single-stream — so batching buys 2.3x total throughput while more than halving each user's speed. That is a normal bandwidth-bound trade, but it is a worse one here than on the NVFP4 build, because there is more weight traffic per token to begin with. If you need concurrency on a Spark, the quantized checkpoint is the lever to pull first, not `--max-num-seqs`.

`pkill -f "vllm serve"` over SSH drops your connection instead of stopping the server.

The remote command's own argv contains the pattern, so pkill matches the shell running it. Use a character class: `pkill -f "[v]llm serve"`. The same trap fires if the *body* of a heredoc you are writing over SSH contains the string `vllm serve` — write the script locally and scp it across instead.