DeepSeek V4 Flash DSpark

median per context · o256

Quick start

1. 1

   Free both nodes first

   This deployment needs ~115 GiB of the 121 GiB on each box, so nothing else can be resident. A leftover Ray cluster or vLLM server from earlier work will silently cost you the KV pool, and the failure arrives ten minutes later as an allocation error deep into startup. Check both nodes before you start.

   for h in 192.168.100.1 192.168.100.2; do
     echo "== $h"
     ssh $h 'free -g | head -2; pgrep -af "vllm|raylet" | head'
   done
   
   # if anything is holding memory:
   ssh <node> '~/venvs/vllm/bin/ray stop --force; pkill -f "vllm serve"'bash

   You want ~117 GiB available and ~5 W idle draw on both nodes before continuing.

2. 2

   Create the deployment directory and compose file (BOTH nodes)

   There is nothing to clone — the recipe is self-contained. Create one working directory on BOTH nodes at the SAME path (start.sh runs docker compose on the worker over SSH from this path), and write docker-compose.yml into it. The compose file pins the runtime image and carries the full vllm serve command; the heredoc is quoted, so paste it verbatim — the $${...} and ${...} are expanded by docker compose at launch, not by your shell.

   # on BOTH nodes — identical path on each
   mkdir -p ~/dspark-v4-flash && cd ~/dspark-v4-flash
   
   cat > docker-compose.yml <<'COMPOSE'
   # DeepSeek V4 Flash DSpark — dual DGX Spark (TP=2, 1M context).
   # Launched by ./start.sh, which injects NODE_RANK, HEADLESS, VLLM_HOST_IP and
   # the per-node NCCL_IB_GID_INDEX. See README.md for why the image is pinned.
   services:
     vllm-dspark:
       image: ${DSPARK_VLLM_IMAGE:-ghcr.io/anemll/dspark-vllm-gx10:0.1.1}
       container_name: ds4-flash-dspark
       pull_policy: if_not_present
       entrypoint: []
       network_mode: host
       ipc: host
       shm_size: "64gb"
       restart: "no"
   
       ulimits:
         memlock: -1 # RDMA registration; without this NCCL silently falls back to TCP
         stack: 67108864
   
       gpus: all
   
       devices:
         - /dev/infiniband:/dev/infiniband
   
       volumes:
         - ${HF_CACHE:-${HOME}/.cache/huggingface}:/cache/huggingface
         - ${DSPARK_TMP_HOST:-${HOME}/.cache/dspark-tmp}:/tmp
   
       environment:
         # ---- model + cache ------------------------------------------------
         HF_HOME: /cache/huggingface
         VLLM_CACHE_ROOT: /cache/huggingface/vllm-cache
         # Default OFF so a first run can download. Flip to 1 once both nodes
         # hold the full 48-shard tree, so a later run can never silently refetch.
         HF_HUB_OFFLINE: "${HF_HUB_OFFLINE:-0}"
         TRANSFORMERS_OFFLINE: "${TRANSFORMERS_OFFLINE:-0}"
         HF_HUB_DISABLE_XET: "1"
   
         # ---- distributed --------------------------------------------------
         NODE_RANK: "${NODE_RANK}"
         HEADLESS: "${HEADLESS:-}"
         MASTER_ADDR: "${MASTER_ADDR}"
         MASTER_PORT: "${MASTER_PORT:-25000}"
         VLLM_HOST_IP: "${VLLM_HOST_IP:-}"
   
         # ---- RoCE fabric ---------------------------------------------------
         # NCCL_IB_GID_INDEX is injected per node by start.sh, which resolves it
         # from sysfs at launch. It differs between nodes and drifts across
         # reboots, so it is deliberately NOT defaulted here.
         NCCL_NET: "IB"
         NCCL_IB_DISABLE: "0"
         NCCL_IB_HCA: "${NCCL_IB_HCA:-rocep1s0f1}"
         NCCL_SOCKET_IFNAME: "${NCCL_SOCKET_IFNAME:-enp1s0f1np1}"
         TP_SOCKET_IFNAME: "${TP_SOCKET_IFNAME:-${NCCL_SOCKET_IFNAME:-enp1s0f1np1}}"
         GLOO_SOCKET_IFNAME: "${GLOO_SOCKET_IFNAME:-${NCCL_SOCKET_IFNAME:-enp1s0f1np1}}"
         NCCL_IB_GID_INDEX: "${NCCL_IB_GID_INDEX:?start.sh must resolve this per node}"
         NCCL_IB_ADDR_FAMILY: "AF_INET"
         NCCL_IB_ROCE_VERSION_NUM: "2"
         NCCL_CROSS_NIC: "1"
         NCCL_CUMEM_ENABLE: "0"
         NCCL_IGNORE_CPU_AFFINITY: "1"
         NCCL_NVLS_ENABLE: "0"
         NCCL_DEBUG: "${NCCL_DEBUG:-WARN}"
   
         # ---- GB10 / sm_121 toolchain ---------------------------------------
         TORCH_CUDA_ARCH_LIST: "12.1a"
         FLASHINFER_CUDA_ARCH_LIST: "12.1a"
         FLASHINFER_DISABLE_VERSION_CHECK: "1"
         FLASHINFER_WORKSPACE_BASE: "/cache/huggingface/flashinfer"
         DG_JIT_USE_NVRTC: "0"
         DG_JIT_NVCC_COMPILER: "/usr/local/cuda/bin/nvcc"
         TILELANG_CLEANUP_TEMP_FILES: "1"
         PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True"
   
         # ---- unified-memory behaviour --------------------------------------
         VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
         VLLM_SKIP_INIT_MEMORY_CHECK: "1"
         VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS: "0"
         VLLM_TRITON_MLA_SPARSE: "1"
         VLLM_SPARSE_INDEXER_MAX_LOGITS_MB: "256"
   
         # ---- DSpark speculative decoding -----------------------------------
         # Clamps stale draft-KV slot ids; required for stable long-context runs.
         DSPARK_SLOT_CLAMP: "1"
         VLLM_USE_FLASHINFER_SAMPLER: "1"
         VLLM_DSPARK_LOCAL_ARGMAX: "1"
         VLLM_DSPARK_REPLICATE_MARKOV_W1: "1"
         VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK: "1"
         VLLM_DSPARK_HARDWARE_SCHEDULER_EARLY_STOP: "1"
         VLLM_DSPARK_CONFIDENCE_THRESHOLD: "0.0"
         VLLM_DSPARK_CONFIDENCE_SCHEDULER: "off"
   
         # ---- b12x FP4 MoE kernels (valid for THIS image only) --------------
         VLLM_USE_B12X_MOE: "1"
         VLLM_USE_B12X_WO_PROJECTION: "1"
         VLLM_B12X_W4A16_FORCE_BLOCKS_MAX_M: "16"
   
         MTP_NUM_TOKENS: "${MTP_NUM_TOKENS:-3}"
   
       command:
         - bash
         - -lc
         - >
           export PATH="/usr/local/cuda/bin:/usr/local/bin:$${PATH:-}";
           export CUDA_HOME="$${CUDA_HOME:-/usr/local/cuda}";
           export LD_LIBRARY_PATH="/usr/local/cuda/lib64:$${LD_LIBRARY_PATH:-}";
           exec /usr/local/bin/vllm serve ${DSPARK_MODEL:-deepseek-ai/DeepSeek-V4-Flash-DSpark}
           --served-model-name ${SERVED_MODEL_NAME:-deepseek-v4-flash-dspark}
           --host ${VLLM_HOST:-0.0.0.0}
           --port ${VLLM_PORT:-8888}
           --trust-remote-code
           --tensor-parallel-size 2
           --pipeline-parallel-size 1
           --kv-cache-dtype nvfp4_ds_mla
           --block-size 256
           --max-model-len ${MAX_MODEL_LEN:-1048576}
           --max-num-seqs ${MAX_NUM_SEQS:-6}
           --max-num-batched-tokens ${MAX_NUM_BATCHED_TOKENS:-8192}
           --max-cudagraph-capture-size $$(( ${MAX_NUM_SEQS:-6} * (${MTP_NUM_TOKENS:-3} + 1) ))
           --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION:-0.85}
           --enable-prefix-caching
           --enable-chunked-prefill
           --async-scheduling
           --speculative-config '{"method":"dspark","num_speculative_tokens":'${MTP_NUM_TOKENS:-3}',"draft_sample_method":"probabilistic"}'
           --tokenizer-mode deepseek_v4
           --distributed-executor-backend mp
           --moe-backend flashinfer_b12x
           --enable-flashinfer-autotune
           --tool-call-parser deepseek_v4
           --enable-auto-tool-choice
           --reasoning-parser deepseek_v4
           --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"<think>","reasoning_end_str":"</think>"}'
           --default-chat-template-kwargs '{"thinking":false}'
           --generation-config vllm
           --nnodes 2
           --node-rank ${NODE_RANK}
           --master-addr ${MASTER_ADDR}
           --master-port ${MASTER_PORT:-25000}
           ${HEADLESS:+--headless}
   COMPOSEbash

   Both nodes need docker-compose.yml at this identical path — start.sh invokes docker compose on the worker over SSH from the same directory. The .env and the scripts in the next steps live on the HEAD only (start.sh ships the .env to the worker for you).

3. 3

   Write the .env (HEAD node only)

   Only the head needs this. start.sh ships it to the worker and overrides the node-specific values (NODE_RANK, HEADLESS, VLLM_HOST_IP, NCCL_IB_GID_INDEX) at the compose call, so edit it here and nowhere else. Set MASTER_ADDR / WORKER_HOST for your pair.

   # on the HEAD node, in ~/dspark-v4-flash
   cat > .env <<'ENVFILE'
   # DeepSeek V4 Flash DSpark — dual DGX Spark (TP2, 1M context)
   #
   # Copy to .env on the HEAD node only. start.sh ships it to the worker and
   # overrides the node-specific values (NODE_RANK, HEADLESS, VLLM_HOST_IP,
   # NCCL_IB_GID_INDEX) at the compose call, so edit it here and nowhere else.
   
   # ---- cluster ---------------------------------------------------------------
   MASTER_ADDR=192.168.100.1
   WORKER_HOST=192.168.100.2
   MASTER_PORT=25000
   # Where this checkout lives on the worker. Blank = same path as the head.
   WORKER_DIR=
   
   # ---- fabric ----------------------------------------------------------------
   # NOTE: NCCL_IB_GID_INDEX is deliberately absent. The RoCEv2 IPv4 GID index
   # differs per node and drifts across reboots (ours: head 3, worker 6), so
   # start.sh resolves it from sysfs on each node at launch. Do not hardcode it.
   NCCL_IB_HCA=rocep1s0f1
   NCCL_SOCKET_IFNAME=enp1s0f1np1
   NCCL_DEBUG=WARN
   
   # ---- model + cache ---------------------------------------------------------
   DSPARK_MODEL=deepseek-ai/DeepSeek-V4-Flash-DSpark
   SERVED_MODEL_NAME=deepseek-v4-flash-dspark
   HF_CACHE=${HOME}/.cache/huggingface
   # Keep 0 until the full 48-shard tree is cached on BOTH nodes, then set 1 so a
   # later run can never silently re-fetch 156 GB.
   HF_HUB_OFFLINE=0
   TRANSFORMERS_OFFLINE=0
   
   # The runtime image. See README for why a stock vLLM cannot serve this
   # checkpoint — this pin is load-bearing, not convenience.
   DSPARK_VLLM_IMAGE=ghcr.io/anemll/dspark-vllm-gx10:0.1.1
   VLLM_PORT=8888
   
   # ---- serving profile -------------------------------------------------------
   # Measured 2026-07-19 on 2x DGX Spark. See deploy README / the recipe page.
   MAX_MODEL_LEN=1048576
   # Leave at 6. The compose file derives --max-cudagraph-capture-size from
   # MAX_NUM_SEQS x (MTP_NUM_TOKENS+1); dropping it to 2 shrinks capture 24 -> 8
   # and costs ~1/3 of decode. This is the one place "small max-num-seqs for a
   # single user" is wrong.
   MAX_NUM_SEQS=6
   MAX_NUM_BATCHED_TOKENS=8192
   GPU_MEMORY_UTILIZATION=0.85
   MTP_NUM_TOKENS=3
   ENVFILE
   
   $EDITOR .env   # set MASTER_ADDR / WORKER_HOST for your pairbash

   Do not set NCCL_IB_GID_INDEX here. start.sh resolves it per node from sysfs at launch, because the index differs between nodes and drifts across reboots.

4. 4

   Write start.sh and stop.sh (HEAD node only)

   These run from the head. start.sh preflights free memory on both nodes, resolves each node's RoCEv2 GID index from sysfs, brings the worker's container up before the head's, then polls for the API. stop.sh tears both down and confirms the memory came back. The orchestration derives from the public MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark; runtime image by Anemll.

   # on the HEAD node, in ~/dspark-v4-flash
   cat > start.sh <<'START'
   #!/usr/bin/env bash
   # Bring up DeepSeek V4 Flash DSpark across two DGX Sparks (TP=2, 1M context).
   # Run from the HEAD node. Starts the worker's container over SSH first, then
   # the head's, then polls for the API.
   #
   # Differences from the upstream MiaAI-Lab scripts this was derived from:
   #   - resolves the RoCEv2 GID index PER NODE from sysfs (they differ between
   #     nodes and drift across reboots; upstream reuses one value and NCCL dies)
   #   - preflights free memory on both nodes (this needs ~115 of 121 GiB each)
   #   - no hardcoded in-image interpreter path
   set -euo pipefail
   
   DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
   cd "$DIR"
   
   [ -f .env ] || { echo "FATAL: no .env here — copy .env.example to .env first."; exit 1; }
   set -a; . ./.env; set +a
   
   : "${MASTER_ADDR:?set in .env}"
   : "${WORKER_HOST:?set in .env}"
   : "${NCCL_IB_HCA:=rocep1s0f1}"
   PROJECT=ds4-flash-dspark
   WORKER_DIR="${WORKER_DIR:-$DIR}"
   COMPOSE=(docker compose -p "$PROJECT" --env-file .env -f docker-compose.yml)
   
   # ---- preflight: both nodes must be essentially empty ------------------------
   # ~78 GiB of weights + ~17 GiB of KV per node leaves no room for a leftover
   # Ray cluster or vLLM server. A stale job shows up ten minutes later as an
   # allocation failure deep into startup, so check now.
   for host in "$MASTER_ADDR" "$WORKER_HOST"; do
     avail=$(ssh "$host" "free -g | awk '/^Mem:/{print \$7}'")
     echo "preflight: $host has ${avail} GiB available"
     if [ "$avail" -lt 100 ]; then
       echo "FATAL: $host has only ${avail} GiB available; need ~115."
       ssh "$host" 'pgrep -af "bin/vllm|raylet" | head' || true
       exit 1
     fi
   done
   
   # ---- resolve each node's RoCEv2 GID index ----------------------------------
   # The IPv4 RoCEv2 GID index is per-node and moves across reboots and link
   # events. Ours were 3 (head) and 6 (worker) simultaneously.
   gid_index() { # $1 = ssh target / fabric ip
     local ip="$1" hex
     hex=$(printf '%02x%02x:%02x%02x' $(echo "$ip" | tr . ' '))
     ssh "$ip" "for g in /sys/class/infiniband/${NCCL_IB_HCA}/ports/1/gids/*; do \
       i=\${g##*/}; \
       [ \"\$(cat /sys/class/infiniband/${NCCL_IB_HCA}/ports/1/gid_attrs/types/\$i 2>/dev/null)\" = 'RoCE v2' ] || continue; \
       case \$(cat \$g) in *ffff:${hex}) echo \$i; break;; esac; done"
   }
   GID_HEAD=$(gid_index "$MASTER_ADDR")
   GID_WORKER=$(gid_index "$WORKER_HOST")
   [ -n "$GID_HEAD" ] && [ -n "$GID_WORKER" ] || {
     echo "FATAL: could not resolve RoCEv2 GID index (head='$GID_HEAD' worker='$GID_WORKER')."
     echo "Check the fabric is up: ibstat | grep -A3 ${NCCL_IB_HCA}"
     exit 1
   }
   echo "RoCEv2 GID index: head=$GID_HEAD worker=$GID_WORKER"
   
   # ---- worker first, then head ------------------------------------------------
   echo "== starting worker on $WORKER_HOST =="
   scp -q .env "$WORKER_HOST:$WORKER_DIR/.env"
   ssh "$WORKER_HOST" "cd $WORKER_DIR && \
     NODE_RANK=1 HEADLESS=1 \
     VLLM_HOST_IP='$WORKER_HOST' NCCL_IB_GID_INDEX='$GID_WORKER' \
     docker compose -p '$PROJECT' --env-file .env -f docker-compose.yml up -d"
   
   echo "== starting head on $MASTER_ADDR =="
   NODE_RANK=0 HEADLESS= \
   VLLM_HOST_IP="$MASTER_ADDR" NCCL_IB_GID_INDEX="$GID_HEAD" \
     "${COMPOSE[@]}" up -d
   
   # ---- wait for the API -------------------------------------------------------
   # Cold start is ~11 min: ~3 min of shard loading per rank, then profiling,
   # flashinfer autotune and cudagraph capture.
   echo "== waiting for API on :${VLLM_PORT:-8888} (cold start ~11 min) =="
   for _ in $(seq 1 120); do
     if curl -sf -m 4 "http://127.0.0.1:${VLLM_PORT:-8888}/v1/models" >/dev/null 2>&1; then
       echo "API is up."
       docker logs "$PROJECT" 2>&1 | grep -E 'Available KV cache|GPU KV cache size|Maximum concurrency' | tail -3 || true
       echo
       echo "Steady-state memory (plan against THIS, not the mid-load figure):"
       for host in "$MASTER_ADDR" "$WORKER_HOST"; do
         ssh "$host" "echo -n '  $host: '; free -g | awk '/^Mem:/{print \$3\" GiB used, \"\$7\" available\"}'"
       done
       echo
       echo "Now warm the model before measuring anything — expert planes fault"
       echo "in from NVMe on first touch. Six full generations is usually enough."
       exit 0
     fi
     sleep 10
   done
   
   echo "FATAL: API did not come up. Recent errors:"
   docker logs --since 25m "$PROJECT" 2>&1 | grep -iE 'error|traceback|failed' | tail -20
   echo
   echo "NOTE: 'docker compose logs' replays PREVIOUS containers of the same"
   echo "project — check timestamps before debugging a stack trace from days ago."
   exit 1
   START
   
   cat > stop.sh <<'STOP'
   #!/usr/bin/env bash
   # Tear down DeepSeek V4 Flash DSpark on both nodes. Run from the HEAD node.
   set -euo pipefail
   
   DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
   cd "$DIR"
   
   [ -f .env ] || { echo "FATAL: no .env here."; exit 1; }
   set -a; . ./.env; set +a
   
   PROJECT=ds4-flash-dspark
   WORKER_DIR="${WORKER_DIR:-$DIR}"
   
   echo "== stopping head =="
   docker compose -p "$PROJECT" --env-file .env -f docker-compose.yml down --remove-orphans || true
   
   echo "== stopping worker on ${WORKER_HOST} =="
   ssh "$WORKER_HOST" "cd $WORKER_DIR && \
     docker compose -p '$PROJECT' --env-file .env -f docker-compose.yml down --remove-orphans" || true
   
   # Confirm the memory actually came back — a half-dead rank holding ~100 GiB is
   # the usual reason the next run fails deep into startup.
   echo
   echo "Memory after teardown:"
   for host in "$MASTER_ADDR" "$WORKER_HOST"; do
     ssh "$host" "echo -n '  $host: '; free -g | awk '/^Mem:/{print \$7\" GiB available\"}'"
   done
   STOP
   
   chmod +x start.sh stop.shbash

5. 5

   Stage the checkpoint on both nodes

   Every TP rank reads all 48 shards to extract its slice, so both nodes need the complete ~156 GB tree. Download once and mirror over the RoCE link rather than pulling twice over the internet — on our pair the worker's default route is wifi, which makes a direct download painfully slow.

   # on the head
   hf download deepseek-ai/DeepSeek-V4-Flash-DSpark
   
   # mirror to the worker over the direct RoCE link
   rsync -a --partial --inplace \
     -e 'ssh -c aes128-gcm@openssh.com -o Compression=no' \
     ~/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-DSpark/ \
     192.168.100.2:~/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-DSpark/bash

   Verify both sides report the same snapshot revision with 48 shards and zero *.incomplete files. Keep HF_HUB_OFFLINE=0 until this is done on both nodes, then set it to 1 so a later run can never silently re-fetch 156 GB.

6. 6

   Resolve each node's RoCEv2 GID index

   The IPv4 RoCEv2 GID index differs per node and moves across reboots and link events. On our pair it is 3 on the head and 6 on the worker simultaneously. Upstream ships a single NCCL_IB_GID_INDEX and reuses it on both nodes, which kills NCCL at init; the env below adds a separate worker value. Re-check this after any reboot.

   for h in 192.168.100.1 192.168.100.2; do
     echo -n "$h -> "
     ssh $h "show_gids | awk '/rocep1s0f1/ && /v2/ && /$h/ {print \$3}'"
   done
   # ours: 192.168.100.1 -> 3 ; 192.168.100.2 -> 6bash

   If this prints nothing, the fabric link is down — check `ibstat` for LinkUp: true before going further.

7. 7

   Review the serving profile

   The values that produced every number on this page are already the defaults in .env.example. The two worth understanding before you change them are MAX_NUM_SEQS and MAX_NUM_BATCHED_TOKENS — both look like obvious single-user tuning targets and both make things worse. See the parameters below.

   # ~/dspark-v4-flash/.env
   MAX_MODEL_LEN=1048576        # the model's own ceiling
   MAX_NUM_SEQS=6               # NOT 1-2 — see keyParams
   MAX_NUM_BATCHED_TOKENS=8192
   GPU_MEMORY_UTILIZATION=0.85
   MTP_NUM_TOKENS=3             # DSpark draft depthbash

8. 8

   Start the cluster from the head

   The script starts the worker's container over SSH first, then the head's, then polls for the API. Expect roughly 11 minutes cold: about 3 minutes of shard loading per rank, then profiling, flashinfer autotune and graph capture.

   cd ~/dspark-v4-flash
   ./start.sh
   
   # watch the numbers that matter
   docker logs -f ds4-flash-dspark 2>&1 \
     | grep -E 'Available KV cache|GPU KV cache size|Maximum concurrency|startup complete'bash

   Expected: 'Available KV cache memory: 17.21 GiB', 'GPU KV cache size: 2,555,830 tokens', 'Maximum concurrency for 1,048,576 tokens per request: 2.44x'.

9. 9

   Confirm steady state, then warm the model

   Check resident memory only after the API answers — this is the figure to plan against. Then warm up: expert planes fault in from NVMe on first touch, so the first requests after a cold start are several times slower than settled behaviour. Ours settle after about six passes.

   for h in 192.168.100.1 192.168.100.2; do ssh $h 'free -g | head -2'; done
   # expect: head ~115 GiB used / ~5 free, worker ~111 / ~9
   
   for i in $(seq 1 6); do
     curl -s http://127.0.0.1:8888/v1/chat/completions \
       -H 'Content-Type: application/json' \
       -d '{"model":"deepseek-v4-flash-dspark",
            "messages":[{"role":"user","content":"Write 200 words about memory bandwidth."}],
            "max_tokens":256,"temperature":0}' \
       -o /dev/null -w "warm $i: %{time_total}s\n"
   donebash

   Stop when consecutive passes agree within a few percent (ours: 5.7-6.4 s). Benchmarking before this point measures NVMe, not the model.

10. 10

    Stop the cluster

    Run from the head; it tears down both nodes. Confirm memory is actually released before starting other work.

    cd ~/dspark-v4-flash
    ./stop.sh
    
    for h in 192.168.100.1 192.168.100.2; do ssh $h 'free -g | head -2'; donebash

Revision history

What has changed on this page since it was published, and what it measured. Newest first.

1. 2026-07-27

   compatibility

   Root-caused why stock vLLM 0.26.0 cannot serve this checkpoint on GB10: two hard-coded block sizes that contradict each other.

   Four TP2 boots sweeping --block-size. 64 is arithmetically impossible (compress_ratio 128 makes storage_block_size 0 -> ZeroDivisionError); 128 is accepted by no backend ('No common block size for 128'); 256 is the ONLY value DeepseekV4IndexerBackend accepts (get_supported_kernel_block_sizes() -> [256]) and is exactly the value flashinfer's sm120 decode kernel refuses (_DECODE_DSV4_PAGE_BLOCK_SIZE = 64), failing the first forward with 'Check failed: num_tokens > 64 (5 vs. 64)'. FLASHINFER_MLA_SPARSE_SM120 is the only sparse-MLA backend on sm_121, so there is nothing to fall back to. Upgrades the previous 'appears to block' finding to a proven one: this is an upstream defect, not a serving config. Also measured: --disable-hybrid-kv-cache-manager clears the old KV-group assertion but costs ~320 KiB/token (40.07 GiB for 131,072 tokens), ~45x this recipe's 7.06 KiB/token. No change to the published serve config.

2. 2026-07-26

   docs

   Made the deployment self-contained — the recipe now inlines docker-compose.yml, .env, start.sh and stop.sh instead of cloning a repo.

   No throughput or serving-config change: the compose file, flags and env are byte-for-byte identical. Only how you obtain the files changed — create ~/dspark-v4-flash on both nodes and paste the four files from the steps, rather than cloning. repoUrl now points at the public MiaAI-Lab upstream the orchestration derives from.

3. 2026-07-26

   correctnessaction needed

   Raised DSpark speculative decoding from k=3 to k=5, the value the checkpoint actually requires.

   The checkpoint declares dspark_block_size 5, so the drafter emits 5-token blocks and k=3 truncated one — vLLM 0.26.0 rejects k<5 configs outright as 'producing incorrect output', while the 0.25.2 image pinned here accepted it silently. Re-measured warm at a matched 8192-token prompt: code 62.5 → 73.2 tok/s (+17.1%), prose 44.2 → 43.1 (−2.5%, flat). Acceptance rate FELL (code 0.847 → 0.825, prose 0.526 → 0.338) while tokens per step ROSE (3.54 → 5.12, 2.58 → 2.69) — tokens/step is what drives throughput.

4. 2026-07-26

   performance

   Trimmed --gpu-memory-utilization from 0.85 to 0.78 to pay for k=5's larger cudagraph capture.

   Capture size is max_num_seqs × (k+1), so it went 24 → 36. KV dropped 17.21 → 8.54 GiB/rank and the pool 2,555,830 → 1,268,110 tokens. No real cost: the pool is still 1.21× the full 1,048,576-token context, and 1M is the model's max_position_embeddings, so surplus pool can never become more context. Steady state 108 GiB head / 105 GiB worker; weights measured at 79.17 GiB/rank.

5. 2026-07-26

   re-measured

   Moved the headline decodeTps to the standard 2048-token prompt (was 8192).

   decodeTps 44.2 @8192 → 42.0 @2048. This is NOT a k=3-vs-k=5 comparison — the measurement context moved at the same time. For the config delta, use the matched-context figures in the correctness entry above.

6. 2026-07-26

   compatibility

   Re-tested stock vLLM 0.26.0 as a replacement for the pinned anemll image. Still blocked, but for a much narrower reason.

   0.26.0 loads the checkpoint fine — the three walls that stopped 0.24.0 (DeepGEMM ue8m0 scales, the DSpark MTP KeyError, CUTLASS sm_121) are all gone, and it auto-selects DEEPGEMM_MXFP4. It dies on a block-size bind instead: flashinfer's sm120 sparse-MLA decode kernel only dispatches at page_block_size 64, but at 64 vLLM's _get_kv_cache_groups_uniform_groups asserts on this model's heterogeneous KV. Also, nvfp4_ds_mla is still image-only, so a stock serve would cost ~1.43× the KV bytes per token. Of the six DeepSeek-V4 speedups in the 0.26.0 notes, three are ROCm-only and one is XPU; the CUDA-relevant three total ~4.7% E2E TPOT.

7. 2026-07-26

   docs

   Documented three operational traps hit while re-measuring.

   The RoCEv2 GID index drifted again (worker 6 → 5, head still 3) and a stale literal in .env.dspark failed ~2 min into startup with 'RuntimeError: NCCL error' — resolve it from sysfs at launch instead of storing it. The loader makes TWO full passes over the 155 GiB tree (main model, then the drafter), so a restarting progress bar is normal and purging page cache before serving is counterproductive once KV is pinned. And `docker compose logs` replays a previous container's traceback into a new run — bound queries with --since and check timestamps.