benchmarks / gemma-4-26b-a4b-it--dgx-spark-x1--vllm--fp8-dynamic--vllm-greedy-baseline--p2048-o256-c1--2026-07-06

Gemma 4 26B-A4B on NVIDIA DGX Spark

compare

39.8tok/s

362msttft

5.6kprefill

vLLM 0.24.0FP8-Dynamicspec-decodingJul 6, 2026

Run configuration

nodes

batch size

—

concurrency

context

10240

prompt / output

2038 / 256

weights RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic ↗

Metrics

decode tok/s

39.8

prefill tok/s

5.6k

ttft

362ms

total tok/s

37.8

inter-token

25ms

e2e latency

6.77s

req/s

0.148

peak memory

70.5 GB

avg power

28.8 W

energy/req

195.1 J

temperature

Reproduce

PATH="$HOME/venvs/vllm/bin:$PATH" VLLM_USE_DEEP_GEMM=0 TORCHINDUCTOR_COMPILE_THREADS=2 MAX_JOBS=4 vllm serve ~/models/hf/gemma-4-26B-A4B-it-FP8-Dynamic --served-model-name gemma-4-26b-a4b-it --max-model-len 10240 --gpu-memory-utilization 0.5 --max-num-seqs 8 --port 8000
python3 bench/harness.py bench/configs/gemma-4-26b-a4b-it--vllm-spec-baseline.json --rows-out data/benchmarks/gemma-4-26b-a4b-it.json

spec-decoding profile: greedy no-draft baseline on vLLM (RedHatAI FP8-Dynamic target).