DeepSeek V4 Flash DSpark

Serve the 284B/13B DeepSeek V4 Flash DSpark checkpoint across two DGX Sparks at its full 1M-token context, with FP4 experts, an NVFP4 MLA KV cache, and DSpark speculative decoding holding decode speed flat out to 275K tokens.

DeepSeek V4 Flash DSpark is a 284B-parameter mixture-of-experts model with 13B active per token, shipped as FP4 routed-expert planes with FP8 attention and dense layers. At 155 GiB on disk it needs two DGX Sparks: tensor parallelism splits it to roughly 78 GiB of weights per node, leaving enough of the 114 GiB usable pool for a KV cache large enough to serve the model's full 1,048,576-token context. Two things make the configuration worth the trouble. First, DSpark speculative decoding — the drafter ships inside the checkpoint, costs about a gibibyte per rank, and roughly doubles single-stream decode. Second, decode speed barely moves as context grows: measured 41.7 tok/s at a 1.4K prompt and 48.3 tok/s at a 275K prompt. What long context actually costs here is time-to-first-token, because prefill runs at 1,400–2,100 tok/s and a 275K-token prompt therefore takes about three and a half minutes before the first token appears. Tuning this deployment is mostly a prefill problem, not a decode one — and the obvious prefill levers turn out to cost more decode than they return, which is documented below.

• 284B total / 13B active MoE — FP4 experts, FP8 attention and dense layers, 155 GiB on disk
• 2 × DGX Spark, tensor-parallel 2 over a direct RoCE link (head 192.168.100.1, worker .2)
• Full 1,048,576-token context — the model's architectural max, not a memory-limited number
• NVFP4 MLA KV cache (nvfp4_ds_mla): 17.21 GiB/rank buys a 2,555,830-token pool
• DSpark speculative decoding, k=3 — drafter ships in the checkpoint, ~1 GiB/rank
• Decode 44–63 tok/s single-stream, workload-dependent (draft acceptance 0.53–0.85)
• Decode holds flat to 275K context; long context costs TTFT (≈199 s at 275K), not tok/s
• Runs on a pinned prebuilt image — a stock vLLM 0.24.0 cannot load this checkpoint at all

Software requirements

• Docker with the compose plugin (v2.24+), on both nodes
• Prebuilt image ghcr.io/anemll/dspark-vllm-gx10:0.1.1 — vLLM 0.25.2.dev0+g752a3a504
• Passwordless SSH from the head to the worker over the RoCE address (192.168.100.2)
• NVIDIA driver 580.159.03 (CUDA 13 capable) — default on DGX Spark
• RoCE tooling: show_gids, ibstat, ibdev2netdev — default on DGX Spark
• ~156 GB free disk per node for the HuggingFace cache
• This deployment config: deploy/deepseek-v4-flash-dspark in the howtospark repo, cloned to the SAME path on both nodes

Quick start

1. 1

   Free both nodes first

   This deployment needs ~115 GiB of the 121 GiB on each box, so nothing else can be resident. A leftover Ray cluster or vLLM server from earlier work will silently cost you the KV pool, and the failure arrives ten minutes later as an allocation error deep into startup. Check both nodes before you start.

   for h in 192.168.100.1 192.168.100.2; do
     echo "== $h"
     ssh $h 'free -g | head -2; pgrep -af "vllm|raylet" | head'
   done
   
   # if anything is holding memory:
   ssh <node> '~/venvs/vllm/bin/ray stop --force; pkill -f "vllm serve"'bash

   You want ~117 GiB available and ~5 W idle draw on both nodes before continuing.

2. 2

   Put the deployment config on both nodes

   The config lives in this site's repo under deploy/deepseek-v4-flash-dspark — a compose file, an env template, and start/stop scripts. Both nodes need it at the SAME path, because the start script invokes docker compose on the worker over SSH from the head's working directory.

   # on BOTH nodes
   git clone https://github.com/jtmuller5/howtospark ~/howtospark
   cd ~/howtospark/deploy/deepseek-v4-flash-dspark
   
   # on the HEAD node only — start.sh ships it to the worker
   cp .env.example .env
   $EDITOR .env   # set MASTER_ADDR / WORKER_HOST for your pairbash

   Do not set NCCL_IB_GID_INDEX in .env. start.sh resolves it per node from sysfs at launch, because the index differs between nodes and drifts across reboots.

3. 3

   Stage the checkpoint on both nodes

   Every TP rank reads all 48 shards to extract its slice, so both nodes need the complete ~156 GB tree. Download once and mirror over the RoCE link rather than pulling twice over the internet — on our pair the worker's default route is wifi, which makes a direct download painfully slow.

   # on the head
   hf download deepseek-ai/DeepSeek-V4-Flash-DSpark
   
   # mirror to the worker over the direct RoCE link
   rsync -a --partial --inplace \
     -e 'ssh -c aes128-gcm@openssh.com -o Compression=no' \
     ~/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-DSpark/ \
     192.168.100.2:~/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-DSpark/bash

   Verify both sides report the same snapshot revision with 48 shards and zero *.incomplete files. Keep HF_HUB_OFFLINE=0 until this is done on both nodes, then set it to 1 so a later run can never silently re-fetch 156 GB.

4. 4

   Resolve each node's RoCEv2 GID index

   The IPv4 RoCEv2 GID index differs per node and moves across reboots and link events. On our pair it is 3 on the head and 6 on the worker simultaneously. Upstream ships a single NCCL_IB_GID_INDEX and reuses it on both nodes, which kills NCCL at init; the env below adds a separate worker value. Re-check this after any reboot.

   for h in 192.168.100.1 192.168.100.2; do
     echo -n "$h -> "
     ssh $h "show_gids | awk '/rocep1s0f1/ && /v2/ && /$h/ {print \$3}'"
   done
   # ours: 192.168.100.1 -> 3 ; 192.168.100.2 -> 6bash

   If this prints nothing, the fabric link is down — check `ibstat` for LinkUp: true before going further.

5. 5

   Review the serving profile

   The values that produced every number on this page are already the defaults in .env.example. The two worth understanding before you change them are MAX_NUM_SEQS and MAX_NUM_BATCHED_TOKENS — both look like obvious single-user tuning targets and both make things worse. See the parameters below.

   # deploy/deepseek-v4-flash-dspark/.env
   MAX_MODEL_LEN=1048576        # the model's own ceiling
   MAX_NUM_SEQS=6               # NOT 1-2 — see keyParams
   MAX_NUM_BATCHED_TOKENS=8192
   GPU_MEMORY_UTILIZATION=0.85
   MTP_NUM_TOKENS=3             # DSpark draft depthbash

6. 6

   Start the cluster from the head

   The script starts the worker's container over SSH first, then the head's, then polls for the API. Expect roughly 11 minutes cold: about 3 minutes of shard loading per rank, then profiling, flashinfer autotune and graph capture.

   cd ~/howtospark/deploy/deepseek-v4-flash-dspark
   ./start.sh
   
   # watch the numbers that matter
   docker logs -f ds4-flash-dspark 2>&1 \
     | grep -E 'Available KV cache|GPU KV cache size|Maximum concurrency|startup complete'bash

   Expected: 'Available KV cache memory: 17.21 GiB', 'GPU KV cache size: 2,555,830 tokens', 'Maximum concurrency for 1,048,576 tokens per request: 2.44x'.

7. 7

   Confirm steady state, then warm the model

   Check resident memory only after the API answers — this is the figure to plan against. Then warm up: expert planes fault in from NVMe on first touch, so the first requests after a cold start are several times slower than settled behaviour. Ours settle after about six passes.

   for h in 192.168.100.1 192.168.100.2; do ssh $h 'free -g | head -2'; done
   # expect: head ~115 GiB used / ~5 free, worker ~111 / ~9
   
   for i in $(seq 1 6); do
     curl -s http://127.0.0.1:8888/v1/chat/completions \
       -H 'Content-Type: application/json' \
       -d '{"model":"deepseek-v4-flash-dspark",
            "messages":[{"role":"user","content":"Write 200 words about memory bandwidth."}],
            "max_tokens":256,"temperature":0}' \
       -o /dev/null -w "warm $i: %{time_total}s\n"
   donebash

   Stop when consecutive passes agree within a few percent (ours: 5.7-6.4 s). Benchmarking before this point measures NVMe, not the model.

8. 8

   Stop the cluster

   Run from the head; it tears down both nodes. Confirm memory is actually released before starting other work.

   cd ~/howtospark/deploy/deepseek-v4-flash-dspark
   ./stop.sh
   
   for h in 192.168.100.1 192.168.100.2; do ssh $h 'free -g | head -2'; donebash

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4-flash-dspark",
       "messages":[{"role":"user","content":"What is 47 x 89?"}],
       "temperature":0, "max_tokens":64}'

# enable the reasoning trace (returned in choices[].message.reasoning_content)
#   "chat_template_kwargs": {"thinking": true}
# Think Max mode wants >= 384K of context to be useful.bash

curl -s http://127.0.0.1:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4-flash-dspark",
       "messages":[{"role":"user","content":"Explain paged attention in detail."}],
       "max_tokens":512, "temperature":0, "stream":true,
       "stream_options":{"include_usage":true}}'

# decode_tps = (completion_tokens - 1) / (t_last_token - t_first_token)
# measured: 44.2 tok/s prose, 62.5 tok/s code, single-stream after warmup (medians of 3 runs each; spread under 2%)bash

curl -s http://127.0.0.1:8888/metrics | grep -E \
  'spec_decode_num_(drafts|draft_tokens|accepted_tokens)_total'

# acceptance   = accepted_tokens / draft_tokens
# tokens/step  = 1 + accepted_tokens / drafts
# measured, per workload:
#   code  : acceptance 0.847, 3.54 tok/step, 17.2 steps/s -> 60.7 tok/s
#   prose : acceptance 0.526, 2.58 tok/step, 17.6 steps/s -> 45.2 tok/sbash

prompt tokens |    TTFT | prefill tok/s | decode tok/s
--------------+---------+---------------+-------------
        1,451 |   1.2 s |         1,260 |        41.7
       22,463 |  10.9 s |         2,061 |        49.5
       91,003 |  50.3 s |         1,807 |        39.5
      274,508 | 198.7 s |         1,382 |        48.3

# measured single-stream, temperature 0, after warmup.
# A 275K-token prompt costs ~3.3 minutes before the first token.text

Troubleshooting

Can a stock vLLM serve this checkpoint instead of the pinned image?

No — we tested it on 2026-07-19 with vLLM 0.24.0 (torch 2.11.0+cu130, TP2 over Ray) and hit three walls, each behind the last. (1) DeepGEMM: 'RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/layout.hpp:59): Unknown SF transformation' — the checkpoint's FP8 dense weights use scale_fmt ue8m0 block scales it cannot lay out. (2) With VLLM_USE_DEEP_GEMM=0 it gets into weight loading and dies at "KeyError: 'model.layers.43.mtp_block.main_norm.weight'" in vllm/models/deepseek_v4/nvidia/mtp.py — 0.24.0's DeepSeek-V4 MTP does not know the DSpark drafter's parameter layout. (3) With speculative decoding off entirely it reaches the first forward and fails in CUTLASS: 'dispatch_scaled_mm, scaled_mm_helper.hpp:17' — no sm_121 kernel for this dtype combination. Separately, 0.24.0 has only fp8_ds_mla (not nvfp4_ds_mla), which roughly halves the KV pool and puts 1M context out of reach, and has no dspark speculative method at all.

Decode drops by a third after lowering --max-num-seqs for single-user serving

The compose file computes --max-cudagraph-capture-size as MAX_NUM_SEQS x (MTP_NUM_TOKENS+1). Setting MAX_NUM_SEQS=2 shrinks capture from 24 to 8, so decode falls off the captured graphs and becomes both slower and much noisier — we measured code 62.5 to 46.8 tok/s and prose 44.2 to 26.0 tok/s, with run-to-run spread widening from a few percent to nearly 2x. The general advice to keep max-num-seqs at 1-2 for one user does not apply when capture size is derived from it. Leave it at 6.

Raising --max-num-batched-tokens to speed up long-context prefill

Tried and reverted. 16384 improved TTFT at a 275K prompt from 199 s to 168 s (~16%) but forced max-num-seqs down to fit memory, costing the decode regression above. The KV pool also shrank from 2,555,830 to 1,570,685 tokens (2.44x to 1.50x concurrency). Net loss for a single user — the startup warning 'max_num_scheduled_tokens is set to 8180 based on the speculative decoding settings' is expected here, not a defect.

NCCL dies at init with 'unhandled system error'

Almost always the RoCEv2 GID index. It differs per node and drifts across reboots — on our pair it is 3 on the head and 6 on the worker at the same time. Upstream ships only NCCL_IB_GID_INDEX and reuses it on both nodes; set WORKER_NCCL_IB_GID_INDEX separately and re-resolve both with `show_gids | grep rocep1s0f1` after any reboot.

'/opt/env/bin/python: no such file or directory' from the helper scripts

The upstream scripts hardcode /opt/env/bin/python, which exists only in the locally built vllm-dspark-runtime image. The prebuilt ghcr.io/anemll image ships /usr/bin/python3. Set DSPARK_PYTHON=/usr/bin/python3.

The first model download fails or hangs

The shipped .env sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1, which breaks a first download. Set both to 0 until the weights are fully cached on BOTH nodes, then flip them back to 1 so a later run can never silently re-fetch.

Stack trace full of c10::Error frames right after starting, but the container is healthy

`docker compose logs` replays the whole history of previous containers with the same project name, so a failure from days ago is re-printed at startup. Check the timestamps — a July-16 'W716' prefix is not today's run. Use `docker logs --since 10m <container>` to see only the current attempt.

Allocation failure late in startup, or a node becomes pingable but SSH-dead

The unified pool was overcommitted; the box is thrashing and userspace, including sshd and tailscaled, is starved. The usual cause is another job still resident — check both nodes for a leftover Ray cluster or vLLM server before starting. Recovery often does not need a power cycle: kill the job on the surviving peer (`ray stop --force`, then `pkill -9 -f 'ray::'`) to release cross-node pressure, and reach that peer over its fabric IP (192.168.100.x), not Tailscale, which will be down on the wedged node.

'RuntimeError: cancelled' in shm_broadcast, or 'Engine core initialization failed'

The head lost a TP worker — this is not a comms bug. Look at the OTHER node's memory and logs; the root cause is almost always there. The traceback names the failing rank's IP (e.g. ip=192.168.100.2).

Throughput looks identical whether the workload is code or prose

You are counting SSE frames rather than tokens. DSpark packs up to 4 tokens per chunk, so frame-counting reports the step rate (~17.5/s), which really is near-constant across workloads. Use stream_options.include_usage and read usage.completion_tokens.

First requests after startup are several times slower than expected

Expert planes fault in from NVMe on first touch. Warm with about six full generations before measuring anything; ours settle to 5.7-6.4 s per 256-token pass. Never benchmark cold.

'free memory less than desired GPU memory utilization' at startup

Page cache is squatting on the unified pool after staging 156 GB of weights. Purge it on BOTH nodes before serving (`~/Dev/vLLM-Moet/spark/purge-cache.py ~/.cache/huggingface`), or simply confirm ~117 GiB available on each node first.