benchmarks / gemma-4-26b-a4b-it--dgx-spark-x1--vllm--fp8-dynamic--vllm-draft-assistant--p2048-o256-c1--2026-07-06
Gemma 4 26B-A4B on NVIDIA DGX Spark
compare53.2tok/s
401msttft
5.1kprefill
vLLM 0.24.0FP8-Dynamicspec-decodingJul 6, 2026
Run configuration
nodes
1
batch size
—
concurrency
1
context
10240
prompt / output
2036 / 256
weights RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic ↗draft model google/gemma-4-26B-A4B-it-assistant ↗
Metrics
decode tok/s
53.2
prefill tok/s
5.1k
ttft
401ms
total tok/s
48.4
inter-token
19ms
e2e latency
5.19s
req/s
0.189
peak memory
69.3 GB
avg power
29.8 W
energy/req
157.6 J
temperature
0
draftAcceptanceRate
0.467
Reproduce
PATH="$HOME/venvs/vllm/bin:$PATH" VLLM_USE_DEEP_GEMM=0 TORCHINDUCTOR_COMPILE_THREADS=2 MAX_JOBS=4 vllm serve ~/models/hf/gemma-4-26B-A4B-it-FP8-Dynamic --served-model-name gemma-4-26b-a4b-it --max-model-len 10240 --gpu-memory-utilization 0.5 --max-num-seqs 8 --port 8000 --speculative-config '{"model": "/home/joemuller/models/hf/gemma-4-assistant", "num_speculative_tokens": 4}'
python3 bench/harness.py bench/configs/gemma-4-26b-a4b-it--vllm-spec-draft.json --scenario 1 --rows-out data/benchmarks/gemma-4-26b-a4b-it.jsonspec-decoding profile: greedy with google gemma-4 assistant (0.42B) via vLLM speculative decoding, 4 speculative tokens.