Solar-Open2 250B NVFP4

2 × Spark1M ctx· memory-bound10.2 tok/s@ 2KvLLMthinking onvanilla

Serve Upstage's Solar-Open2 (250B-class hybrid linear-attention MoE, 320 experts / top-8, ~8B active) in Nota's NVFP4 build across two DGX Sparks (TP2) — 10.2 tok/s single-stream decode, serving the full 1M native context (NoPE, no rope scaling), decode-bound by the BF16 attention the quant leaves untouched. Runs on a hand-written out-of-tree vLLM model plugin (no released engine implements the architecture).

Solar-Open2 is a hybrid linear-attention MoE: 48 layers, hidden 4096, ~145B parameters, 320 routed experts + 1 shared (top-8, ~8B active per token). Only 12 of 48 layers are full-attention GQA (NoPE, sigmoid output gate); the other 36 are KDA gated-delta linear-attention layers with constant per-layer state, so the KV cache is ~1/4 of a dense-attention 48-layer model — which is what lets the model's full native 1M context fit on two Sparks. Because positional information lives in the linear layers' recurrent state (NoPE), 1M is a real native window, not a rope-scaling stretch. The serving difficulty is not context — it is that no released engine implements the `solar_open2` architecture, so this recipe runs a hand-written out-of-tree vLLM model plugin (assembled from vLLM's existing Kimi-Delta-Attention, Qwen3-Next gated-GQA, and DeepSeek-V3 MoE parts). Nota's NVFP4 build quantizes only the routed experts and lands at ~153 GB, which TP2 splits to 71.48 GiB/rank. A MEASURED 28 GiB KV pool holds 1,133,999 tokens, so the full 1,048,576 window serves at 1.08x — but it runs the box near its ceiling (~8 GiB free on the head node, right at the oom-guard line): stable for short prompts, tight for a full-length one. Dropping to an 8 GiB pool serves 256K at 1.22x with ~29 GiB/node to spare, if you want margin over reach. Tuned for a single user: 10.2 tok/s single-stream decode (at a 2048-token prompt). That decode number is low for the model's active size because it is bandwidth-bound by the BF16 attention the quant deliberately leaves untouched — measured ~10 tok/s is ~67% of the ~15 tok/s roofline, and there is no MTP/draft in the checkpoint to speed it up. Decode falls gently as context fills (10.6 tok/s at 8K prompt, 8.1 at ~223K) while TTFT climbs steeply (4 s at 8K, 162 s at ~223K, ~14 min extrapolated at 1M) — long context is cheap in memory but the prefill is compute-bound, so a very long prompt costs minutes before the first token. Read the troubleshooting section first: the one non-obvious failure — a `profile_run` OOM that is really a JIT-compile memory burst — costs an ~11-minute boot to rediscover.

• 2 × DGX Spark (GB10, sm_121), tensor-parallel (TP2) — the full ~145B model, no prune
• The full 1M native context serves (verified: 28 GiB pool -> 1,133,999 tokens, 1.08x at 1,048,576) — NoPE-native, no rope scaling
• 1M runs near the memory ceiling: ~8 GiB free on the head node (at the oom-guard line). For margin, an 8 GiB pool serves 256K at 1.22x with ~29 GiB/node free
• No released engine implements `solar_open2` — served via a hand-written out-of-tree vLLM plugin (vllm/solar_open2/)
• Hybrid attention: 12/48 layers are full-attn GQA (NoPE), 36/48 are KDA linear-attention with constant state -> KV is cheap (why 1M fits)
• NVFP4 on the 320 routed experts only; ALL attention, MoE gates, the shared expert and lm_head stay BF16
• Measured: 71.48 GiB weights per rank; the profiling forward adds only ~0.3 GiB (the profile_run OOM is a JIT-compile burst, not the forward)
• Decode is attention-bound: BF16 attention is 8.2% of the file but ~68% of per-token reads -> ~15 tok/s roofline
• Measured: 10.2 tok/s single-stream decode at 2048-prompt, ~67% of the roofline (--enforce-eager + cross-node TP2 cost the rest)
• Long context is cheap in memory but not in prefill: TTFT 4 s at 8K -> 162 s at ~223K (~14 min extrapolated at 1M); decode eases 10.6 -> 8.1 tok/s
• No MTP/draft in the checkpoint, and vLLM's spec-decode path asserts on the KDA linear-attn state -> no speculative-decode lever without further port work
• The one trap: MAX_JOBS=4 on the raylets, or the flashinfer NVFP4-MoE JIT OOMs the worker during profiling

Software requirements

• 2 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.25.1 + torch 2.11.0+cu130 in a venv (~/venvs/vllm-025), CUDA 13 (required for the NVFP4 MoE kernels)
• The out-of-tree solar_open2 plugin installed into the venv on BOTH nodes (vllm/solar_open2/install.sh) — no released vLLM registers this architecture
• ninja (~/.local/bin) + nvcc (/usr/local/cuda/bin) on the raylet PATH — the flashinfer NVFP4-MoE kernel JIT-compiles on the first forward
• Direct RoCE link between nodes with static point-to-point IPs; resolve each node's RoCEv2 GID index at start time (differs per node, drifts across reboots)
• ~153 GB free disk per node — both nodes need the full 29-shard checkpoint at the same path
• Passwordless SSH between nodes; loginctl enable-linger on both (else raylets die on SSH logout)

Quick start

1. 1

   Install the out-of-tree solar_open2 plugin on both nodes

   No released vLLM implements `solar_open2` / `SolarOpen2ForCausalLM`, and --trust-remote-code only wires up config/tokenizer, not the forward pass. The plugin (in this repo at vllm/solar_open2/) assembles the model from vLLM's existing parts: Kimi-Delta-Attention for the 36 linear layers, a NoPE gated-GQA attention for the 12 full-attn layers, and a DeepSeek-V3-style FusedMoE. install.sh copies the model + config into the venv in-tree (with .orig backups) and registers the arch. Each node has its own venv, so run it on BOTH.

   # on BOTH nodes
   bash vllm/solar_open2/install.sh ~/venvs/vllm-025bash

   Two correctness points are baked into the port and must not regress: the KDA delta-rule beta is doubled (kda_allow_neg_eigval=true, which vLLM's Kimi path does not handle), and q/k/v are built as SEPARATE projections (not fused) so each matches the checkpoint's per-projection NVFP4 ignore entry and stays BF16.

2. 2

   Stage the checkpoint on both nodes (same path)

   Each TP rank reads all shards to extract its slice, so both nodes need the complete ~153 GB tree at an identical path. Download once, then mirror over the direct RoCE link (Spark-2's default route is wifi, so a direct download there is slow).

   # node 1
   hf download nota-ai/Solar-Open2-250B-Nota-NVFP4 \
     --local-dir ~/models/hf/Solar-Open2-250B-Nota-NVFP4
   
   # mirror to node 2 over the direct RoCE link
   rsync -a --partial --inplace \
     -e 'ssh -c aes128-gcm@openssh.com -o Compression=no' \
     ~/models/hf/Solar-Open2-250B-Nota-NVFP4/ \
     192.168.100.2:~/models/hf/Solar-Open2-250B-Nota-NVFP4/bash

   Verify both sides end up with 29 *.safetensors shards and zero *.incomplete files before serving — a partial tree fails deep into load.

3. 3

   Start the Ray cluster with MAX_JOBS bounded (head first, then worker)

   TP2 is placed by Ray, so the cluster comes up before vLLM. The load-bearing addition for THIS model is MAX_JOBS=4 on the raylet env: the NVFP4-MoE backend (FLASHINFER_CUTLASS, auto-selected on GB10) JIT-compiles an sm120 cutlass kernel on the first forward — which happens inside vLLM's profiling — and ninja defaults to one nvcc per core (20 on a Spark), each a few GB, which OOMs a worker already holding 71.48 GiB of weights. Ray actors inherit the RAYLET's env, so MAX_JOBS (and the RoCE fabric env) must be set here, not in the serve script. Resolve each node's RoCEv2 GID index at start time; they differ per node and drift across reboots.

   # on the head node (Spark-1). MAX_JOBS bounds the flashinfer JIT compile memory.
   MAX_JOBS=4 bash ~/solar_plugin/start-ray-cluster-025.sh
   # expect: 2 nodes, 0.0/2.0 GPUbash

   loginctl enable-linger $USER must be set on BOTH nodes, or a self-SSH inside the serve path tears the user manager down and kills the worker raylet (surfaces as vLLM hanging in 'Waiting for creating a placement group').

4. 4

   Serve TP2, --enforce-eager, with an explicit KV pool

   --enforce-eager is required: cross-node TP2 cudagraph replay deadlocks on GB10 (the first generation hangs with the head GPU pinned and the worker idle). Hybrid attention makes KV cheap, so set it explicitly with --kv-cache-memory-bytes. A 28 GiB pool reaches the full 1M window at 1.08x — but leaves the head node at ~8 GiB free (the oom-guard line), so a full 1M-token prefill runs it tight; for comfortable margin use an 8 GiB pool and --max-model-len 262144 (256K at 1.22x, ~29 GiB/node free). Purge the page cache on both nodes first (the serve script does this).

   # on the head node. 28 GiB KV pool -> 1,133,999 tokens -> full 1,048,576 (1M) at 1.08x.
   # For margin instead: KV_BYTES=8000000000 MAXLEN=262144  (256K at 1.22x, ~29 GiB free).
   setsid nohup env \
     MAXLEN=1048576 GPU_UTIL=0.72 KV_BYTES=28000000000 \
     MAX_NUM_SEQS=1 MAX_BATCHED=2048 \
     bash ~/solar_plugin/serve-solar-open2-tp2.sh \
     >/tmp/solar_serve.log 2>&1 </dev/null &
   
   # watch (first boot ~11 min: 29 shards + one-time flashinfer JIT + profiling;
   #        later boots ~2 min once the kernel is cached under ~/.cache/flashinfer)
   tail -f /tmp/solar_serve.log   # or: curl localhost:8000/healthbash

   The serve script sets --served-model-name solar-open2, --distributed-executor-backend ray, --tensor-parallel-size 2, --enforce-eager. At 28 GiB / 1M do NOT push the pool higher: 32 GiB sized 1.29M tokens but tripped the oom-guard by 18 MiB during allocation. Confirm the log shows `Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend` and `Model loading took 71.48 GiB` on both ranks.

5. 5

   Smoke test, then read the KV pool you actually got

   Confirm coherence (this also validates the port's numerics — a wrong KDA delta-rule would produce garbage, not clean facts), then read the KV numbers from the startup log and the steady-state memory.

   curl -s localhost:8000/v1/completions \
     -H 'Content-Type: application/json' \
     -d '{"model":"solar-open2","prompt":"The capital of Japan is","max_tokens":8,"temperature":0}'
   # -> " Tokyo. Tokyo is the largest city in"
   
   grep -E 'GPU KV cache size|Maximum concurrency' /tmp/solar_serve.log
   # GPU KV cache size: 1,133,999 tokens
   # Maximum concurrency for 1,048,576 tokens per request: 1.08x
   
   # steady-state memory — at 1M the head sits near the oom-guard line
   free -g | sed -n '2p'          # head: ~113 used / ~8 available
   ssh 192.168.100.2 "free -g | sed -n '2p'"   # worker: ~109 / ~12bash

   Warm the model with a couple of throwaway generations before benchmarking — the first forward pays the JIT + expert-plane faults (an 8s-timeout curl right after startup returns nothing; give the first request 60s). The model emits a 'Thinking Process:' preamble (it is reasoning-style).

Key vLLM parameters

API usage

curl -s localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"solar-open2",
       "prompt":"Write a detailed technical essay about the history of computing.",
       "max_tokens":256, "temperature":0, "stream":true,
       "stream_options":{"include_usage":true}}'

# decode_tps = (completion_tokens - 1) / (t_last_token - t_first_token)
# measured: 10.2 tok/s single-stream at 2048-token promptbash

# send a long prompt; expect minutes of TTFT before the first token at >100K.
curl -s localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"solar-open2","prompt":"<very long context>\n\nSummarize.",
       "max_tokens":128, "temperature":0, "stream":true,
       "stream_options":{"include_usage":true}}'
# measured at ~223K: TTFT 162 s, then decode 8.1 tok/sbash

Troubleshooting

profile_run OOMs the worker (`killed by the OS (out of memory)`, head logs `RuntimeError: cancelled`) — looks like the profiling forward allocating ~35 GiB beyond the weights

It is NOT the forward — the profiling forward costs ~0.3 GiB (measured per-layer). The tens of GB is the FLASHINFER_CUTLASS NVFP4-MoE backend JIT-compiling its sm120 cutlass kernel on the first forward (inside profile_run): ninja runs nproc(=20) parallel nvcc, ~3-4 GiB each, on a worker already holding 71.48 GiB of weights. The tell is `pgrep -c 'cicc|nvcc|ptxas'` spiking on the WORKER during profiling — the compiler memory never shows up in torch.cuda.* accounting. Fix: MAX_JOBS=4 on the raylet env (with it, the worker held ~36 GiB free and sized the KV pool cleanly). Do NOT chase this by shrinking --max-model-len / --max-num-seqs / --max-num-batched-tokens; the forward is cheap, bound the compiler.

oom-guard kills the server during KV allocation at 1M (`TRIP: MemAvailable ... < 8192 MiB`)

The 1M window at 28 GiB KV leaves the head node at only ~8 GiB free — right at the guard line. A larger pool overshoots: 32 GiB sized 1,296,304 tokens (1.24x) but MemAvailable dipped to 8174 MiB during allocation and the guard SIGKILLed it. Stay at 28 GiB for 1M, or drop to an 8 GiB pool + --max-model-len 262144 for 256K with ~29 GiB/node of margin. The guard kills cleanly (no box wedge), so a miss costs a boot, not a power cycle.

`FileNotFoundError: [Errno 2] No such file or directory: 'ninja'` deep inside gen_cutlass_fused_moe_sm120_module().build_and_load()

The flashinfer MoE JIT shells out to ninja (and nvcc). Ray raylets inherit the LOGIN-shell PATH, which has ninja (~/.local/bin) and nvcc (/usr/local/cuda/bin); a non-login launch does not. Ensure the raylets are started from a login shell (the cluster script self-SSHs to get one) and that both tools are installed on each node.

First generation hangs — head GPU at ~96%, worker at 0% — then `TimeoutError: RPC call to sample_tokens timed out`

Cross-node TP2 cudagraph-replay deadlock on GB10. NCCL_GRAPH_MIXING_SUPPORT alone is not enough over the RoCE link. Serve with --enforce-eager (the recipe does). It costs decode but is deterministic.

Engine core initialization failed: `No node info found matching attributes` / `Failed to connect to Ray cluster`

A stale/degraded Ray cluster — usually after a previous run was SIGKILLed (e.g. by the oom-guard), which can leave the placement machinery broken even though `ray status` still shows 2 nodes. Restart the cluster cleanly: `ray stop --force` on both nodes, kill leftover EngineCore procs (by PID or a `[E]ngineCore` char-class — a plain `pgrep -f EngineCore` self-matches your own command), then re-run start-ray-cluster-025.sh before serving.

139 `Skipping unknown weight` warnings for full-attn q/k/v and the shared expert's gate/up, then load fails

A projection-fusion bug: the NVFP4 checkpoint's ignore list names each projection SEPARATELY (q_proj/k_proj/v_proj/gate_proj/up_proj, all BF16), so a fused qkv_proj/gate_up_proj matches no ignore entry, gets NVFP4-quantized, and then the BF16 checkpoint weight has no matching packed param. The plugin builds these as separate ColumnParallelLinear so each prefix matches its ignore entry. If you see this, a fusion crept back in.

Coherent-looking but subtly wrong output (plausible garbage)

Check the KDA beta doubling. Solar sets kda_allow_neg_eigval=true (beta in [0,2]); vLLM's Kimi KDA path assumes beta in [0,1] and does not double it. SolarOpen2LinearAttention.forward applies beta *= 2. Missing it produces plausible garbage, not a crash — validate against a factual prompt (`The capital of Japan is` -> ` Tokyo.`).

vLLM hangs in `Waiting for creating a placement group` (only 1 GPU registers)

A systemd --user service/scope only lives as long as the login session unless `loginctl enable-linger $USER` is set. A self-SSH inside the serve path then tears the user manager down and kills the worker raylet. Set enable-linger on BOTH nodes once.

A very long prompt takes minutes before the first token

Expected. Long context is cheap in KV memory here (hybrid attn) but prefill is compute-bound: measured TTFT is 4 s at 8K, 16 s at 32K, 81 s at 128K, and 162 s at ~223K tokens (prefill ~2050 tok/s falling to ~1370 at length); a full 1M-token prompt extrapolates to ~14 min of prefill. Decode over the measured range eases from 10.6 to 8.1 tok/s. If you need fast first-token at length, that is a hardware limit, not a misconfiguration.

Enabling speculative decode (ngram/EAGLE/draft) crashes on the first token with `assert non_spec_state_indices_tensor is not None`

vLLM's spec-decode path is not wired for this model's KDA linear-attention. It initializes (engine logs `SpeculativeConfig(method='ngram', ...)`) but dies fatally on the first generated token in kimi_gdn_linear_attn.py: the KDA recurrent state must be partitioned into speculative vs non-speculative slots, which the port's attention-metadata builder doesn't populate. This blocks EVERY spec method (ngram, EAGLE, draft, MTP), not just ngram — so there is no spec-decode speed lever without further vLLM-internals port work. The ngram config fails fast and clean (no box wedge), so it's safe to confirm.