Gemma 4 26B-A4B — FP8-Dynamic

No measured runs yet — be the first: install the spark-benchmark skill.

Quick start

1. 1

   Free the node first

   Anything still holding unified memory — a previous vLLM, a leftover Ray cluster — comes out of the same pool.

   pgrep -af 'vllm serve|ray::' || echo 'node is clear'
   ray stop --force 2>/dev/null || true
   free -gbash

2. 2

   Download the target and the draft

   The target is a single ~27 GB model.safetensors, not a sharded export, so the download is one long file rather than many. The draft is small enough to be an afterthought.

   hf download RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic \
     --local-dir ~/models/hf/gemma-4-26B-A4B-it-FP8-Dynamic
   
   hf download google/gemma-4-26B-A4B-it-assistant \
     --local-dir ~/models/hf/gemma-4-assistantbash

   Check for leftover *.incomplete files under .cache/huggingface/download/ before serving — a single-file checkpoint gives you no shard count to sanity-check against.

3. 3

   Confirm the context is as cheap as it looks

   Before budgeting memory, read the layer types out of the config. This is the fact the whole recipe rests on: only the full_attention layers keep a global KV cache, and here there are five of them.

   python3 - <<'PY'
   import json, collections
   c = json.load(open('/home/joemuller/models/hf/gemma-4-26B-A4B-it-FP8-Dynamic/config.json'))
   t = c['text_config']
   print(collections.Counter(t['layer_types']))
   print('sliding window        :', t['sliding_window'])
   print('global kv heads       :', t['num_global_key_value_heads'])
   print('max position embeddings:', t['max_position_embeddings'])
   PYbash

   Prints Counter({'sliding_attention': 25, 'full_attention': 5}), window 1024, 2 global KV heads, 262144 positions.

4. 4

   Serve it, with the quantized draft, on vLLM 0.24.0

   Two things need explaining. The KV pin: left to itself the engine takes 75 GiB of cache and reports 12x concurrency, which a single user cannot spend — 7 GiB still admits a full-length request and gives the rest of the machine back. And the draft: 0.24.0 cannot load a quantized Gemma 4 MTP draft as shipped, so the one-file patch below is not optional. Without it the draft loads silently with random projections and acceptance falls to ~3%.

   python3 scripts/patch_gemma4_mtp_draft_quant.py --venv ~/venvs/vllm
   
   PATH="$HOME/venvs/vllm/bin:$PATH" \
   vllm serve ~/models/hf/gemma-4-26B-A4B-it-FP8-Dynamic \
     --served-model-name gemma-4-26b-a4b-it \
     --max-model-len 262144 \
     --gpu-memory-utilization 0.85 \
     --kv-cache-memory-bytes 7516192768 \
     --max-num-seqs 4 \
     --port 8000 \
     --speculative-config '{"model": "'$HOME'/models/hf/Sparkulator-Gemma-4-26B-A4B", "num_speculative_tokens": 4}'bash

   No VLLM_USE_DEEP_GEMM=0 is needed for this checkpoint — vLLM selects the TRITON FP8 MoE backend on its own. The draft is sapidlabs/Sparkulator-Gemma-4-26B-A4B; hf download it first. Serving the stock BF16 draft instead is fine and needs no patch — it just costs 10% of decode.

5. 5

   Confirm what it actually chose

   Three lines tell you the configuration came up as intended: the MoE backend, the resident weight size, and how much context the pinned cache actually admits.

   grep -E "Fp8 MoE backend|Model loading took|GPU KV cache size|Maximum concurrency" /tmp/vllm.logbash

   On vLLM 0.24.0 expect TRITON, 26.61 GiB, 323,358 tokens and 1.23x — all four confirmed 2026-07-27. 25.83 GiB instead of 26.61 means the draft did not load.

6. 6

   Warm it up before you trust any number

   The first requests after load pay CUDA-graph and speculative-decode JIT costs. Three are enough here.

   for i in 1 2 3; do
     curl -s http://127.0.0.1:8000/v1/chat/completions \
       -H 'Content-Type: application/json' \
       -d '{"model":"gemma-4-26b-a4b-it",
            "messages":[{"role":"user","content":"Write an LRU cache in Python."}],
            "max_tokens":256,"temperature":0}' > /dev/null
   donebash

7. 7

   Read the acceptance rate, not just the tok/s

   vLLM does not put draft counts in the response stream, so they have to come from the Prometheus endpoint. The counters are cumulative for the life of the server — snapshot before and after a workload and subtract.

   curl -s localhost:8000/metrics | grep -E '^vllm:spec_decode' | grep -v '^#'
   # accepted_tokens_total / draft_tokens_total          = mean acceptance
   # accepted_tokens_per_pos_total{position=N} / drafts  = acceptance at position Nbash

   Measured over the benchmark window on 2026-07-27: 5,100 drafted, 1,895 accepted = 37.2% mean, and 52.9 / 35.7 / 31.5 / 28.5% at positions 0-3. Position 3 still lands more than a quarter of the time, which is why k=4 is worth its cost here.

Revision history

What has changed on this page since it was published, and what it measured. Newest first.

1. 2026-07-27

   docs

   Fixed the data shape of the highlights and software-requirements lists: they were authored as {label, value} objects where the schema is a list of strings, which crashed the recipe page server-side. Flattened each entry into one sentence; no serving flags, measurements or memory figures changed.

2. 2026-07-27

   performance

   Switched the draft to sapidlabs/Sparkulator-Gemma-4-26B-A4B, a W4A16 quantization of Google's assistant draft, and added the vLLM patch it needs to load at all.

   Decode at a 2,048-token prompt 66.8 → 73.9 tok/s (+10.7%) on Spark-1, and 66.5 → 73.0 (+9.9%) on Spark-2 running the same pair in the reverse order; 512-token prompt 94.5 → 99.1 and 92.1 → 100.0; 8,192-token prompt 42.2 → 43.7 and 40.8 → 45.0. Six of six counterbalanced arms favour the quantized draft. Draft on disk 0.782 → 0.585 GiB, resident weights 26.61 → 26.41 GiB. Acceptance on a fixed prompt set 62.8% vs 60.5–65.3% for the BF16 draft, i.e. unchanged — and speculative decoding is verified, so output is identical either way. The published 63.4 tok/s headline was an earlier session's BF16-draft measurement; the matched in-session BF16 baseline for this A/B was 66.8.