benchmarks / nemotron-3-puzzle-75b-a9b--dgx-spark-x1--vllm--nvfp4--p2048-o256-c1--2026-07-07

Nemotron 3 Puzzle 75B-A9B on NVIDIA DGX Spark

compare

20.2tok/s

631msttft

3.2kprefill

vLLM 0.24.0NVFP4Jul 7, 2026

Run configuration

nodes

batch size

—

concurrency

context

10240

prompt / output

2043 / 256

weights nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 ↗

Metrics

decode tok/s

20.2

prefill tok/s

3.2k

ttft

631ms

total tok/s

19.3

inter-token

49ms

e2e latency

13.24s

req/s

0.076

peak memory

82.1 GB

avg power

30.5 W

energy/req

403.6 J

Reproduce

PATH="$HOME/venvs/vllm/bin:$PATH" VLLM_USE_DEEP_GEMM=0 TORCHINDUCTOR_COMPILE_THREADS=2 MAX_JOBS=4 vllm serve ~/models/hf/Nemotron-Puzzle-75B-A9B-NVFP4 --served-model-name Nemotron-Puzzle-75B-A9B-NVFP4 --max-model-len 10240 --gpu-memory-utilization 0.6 --max-num-seqs 8 --limit-mm-per-prompt '{"image": 0, "video": 0}' --trust-remote-code --port 8000
python3 bench/harness.py bench/configs/nemotron-3-puzzle-75b-a9b-nvfp4--vllm.json --rows-out data/benchmarks/nemotron-3-puzzle-75b-a9b.json

NVIDIA NVFP4 of Nemotron-3-Puzzle (nemotron_h_puzzle hybrid Mamba MoE, ~9B active, 75B total). Native Blackwell FP4 on GB10.