Ornith 1.0 35B — FP8

Deep Reinforce's agentic-coding Ornith 1.0 35B (35B MoE, ~3B active) in vendor FP8 on one DGX Spark, serving the full 262,144-token context at 37.7 tok/s single-stream.

Ornith 1.0 35B is Deep Reinforce's agentic-coding model, post-trained on the Qwen3.5 35B-A3B MoE stack: 256 routed experts at top-8, roughly 3B active parameters per token, 30 of its 40 layers gated-delta linear attention and 10 full attention, a 262,144-token native window, and MIT licensed. This recipe serves the vendor's own FP8 export of it, which is the configuration to reach for first if you want the model fast rather than bit-exact. It is a W8A8 compressed-tensors cut with per-channel weight scales and dynamic per-token activation scales, and what matters about it is not the file size but the coverage: the ignore list is only six patterns long, so both halves of the per-token read path get compressed — the 256 routed experts that are 92% of the parameters, and the q/k/v/o projections on the ten full-attention layers. That is why the speedup is real rather than cosmetic. Decode on the Spark is bandwidth-bound, and a quantization that halves the bytes a token reads shows up directly in tokens per second: measured 37.7 tok/s single-stream at a 2048-token prompt, against 30.3 for the same model, same node and same flags in BF16. Prefill gains too, by a wider margin — 5,096 tok/s at 2048 against 3,746 — because prefill is compute-bound and the FP8 tensor cores are doing that arithmetic. Memory is the other half of the story and it is arguably the bigger one. The weights land at 34.87 GiB where BF16 needs 65.53, so with an 8 GiB KV pin the whole serving working set is about 51 of 114 usable GiB. More than half the machine is free while serving the full context, which is what makes this the version to run if you also want a draft model resident, a second service on the box, or simply headroom you do not have to think about. Two caveats before you copy the serve command. There is no speculative draft: the config advertises mtp_num_hidden_layers: 1, but the weight index holds 62,546 tensors and not one matches 'mtp'. And this is a genuinely multimodal release whose vision tower is left in BF16 by the quantization — all 333 visual.* tensors are present, so it loads on stock vLLM with no patching and must not be served with --language-model-only.

• 1 x DGX Spark (GB10, sm_121) — no tensor parallelism, no second node
• Measured 37.7 tok/s single-stream decode at a 2048-token prompt, warm, greedy
• Weights 34.87 GiB against 65.53 GiB for the same model in BF16 — the working set is ~51 of 114 usable GiB, so over half the Spark stays free
• Full 262,144-token context served, at 1.58x concurrency on an 8 GiB pinned KV pool
• The claimed context is proven, not assumed: a single 261,224-token prompt served at 20.9 tok/s decode after 113 s of prefill
• The vendor's own FP8 export: W8A8 compressed-tensors, per-channel weights, dynamic per-token activations, only six ignore patterns
• Both halves of the per-token read path are quantized — the 256 routed experts AND the full-attention q/k/v/o — which is why decode moves and not just the file size
• Prefill gains more than decode: 5,096 tok/s at a 2048-token prompt, 6,062 at 8K
• Decode falls gently with prompt length (37.9 / 37.7 / 36.9 / 34.3 / 26.5 / 21.0 tok/s at 0.5K / 2K / 8K / 32K / 131K / 250K) while TTFT climbs from 0.19 s to 106 s
• Loads on stock vLLM 0.24.0 with no patches and no --language-model-only: all 333 vision-tower tensors are present and left in BF16
• vLLM picks the TRITON FP8 MoE backend on sm_121 — the FlashInfer FP8 paths are offered but not selected
• No speculative draft available — the config advertises mtp_num_hidden_layers: 1 but the checkpoint ships zero MTP tensors
• Weight load ~233 s cold from NVMe, then ~41 s of profiling, compile and warmup

Software requirements

• 1 x DGX Spark (GB10, sm_121), NVIDIA driver >= 580 (CUDA-13 capable)
• vLLM 0.24.0 with torch 2.11.0+cu130 (last verified 2026-07-27 on 0.24.0). The architecture Qwen3_5MoeForConditionalGeneration is also registered in vLLM 0.25.1 and 0.26.0, but the FP8 MoE backend menu differs between releases on sm_121 — re-measure rather than assuming
• The `hf` CLI to fetch the checkpoint (~37.7 GB)
• ~45 GB of free disk on the serving node

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ornith-fp8",
    "messages": [{"role": "user", "content": "Explain MoE routing in three sentences."}],
    "max_tokens": 512,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | jq -r '.choices[0].message.content'bash

import json, time, urllib.request, uuid

body = {
    "model": "ornith-fp8",
    "messages": [{"role": "user",
                  "content": f"[{uuid.uuid4().hex[:8]}] Write an LRU cache in Python."}],
    "max_tokens": 256, "temperature": 0, "stream": True,
    "stream_options": {"include_usage": True}, "ignore_eos": True,
    "chat_template_kwargs": {"enable_thinking": False},
}
req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions",
                             data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
t0 = time.perf_counter(); t_first = None; usage = None
with urllib.request.urlopen(req) as res:
    for raw in res:
        line = raw.decode().strip()
        if not line.startswith("data: "):
            continue
        if line[6:] == "[DONE]":
            break
        ch = json.loads(line[6:])
        usage = ch.get("usage") or usage
        if ch.get("choices") and ch["choices"][0]["delta"].get("content") and not t_first:
            t_first = time.perf_counter()
t_end = time.perf_counter()
print(f"ttft {1000*(t_first-t0):.0f} ms")
print(f"decode {(usage['completion_tokens']-1)/(t_end-t_first):.1f} tok/s")python

python3 -c "import json; q=json.load(open('$HOME/models/hf/Ornith-1.0-35B-FP8/config.json'))['quantization_config']; print(q['quant_method'], q['config_groups']['config_group_0']['format'] if 'format' in q['config_groups']['config_group_0'] else q['format']); print('\n'.join(q['ignore']))"
# compressed-tensors float-quantized
# lm_head
# re:.*embed_tokens.*
# re:.*\.mlp\.gate$
# re:.*shared_expert_gate$
# re:.*linear_attn.*
# re:model\.visual\..*bash

Troubleshooting

Every answer starts with 'Here's a thinking process: 1. Understand User Request...' before any code.

That is the model reasoning, not a template bug. Ornith's chat_template.jinja opens the assistant turn with '<think>\n' unless the request passes enable_thinking: false, in which case it emits an empty '<think>\n\n</think>\n\n' pair and the answer starts immediately. Send it as "chat_template_kwargs": {"enable_thinking": false}. Decide this before you benchmark: with thinking on, most of the generated tokens are reasoning, so the throughput numbers are not comparable with the ones here.

The MoE backend line says TRITON, not FlashInfer. Did something fall back?

No — that is the correct auto-selection for this checkpoint on sm_121 and it is the configuration all the numbers here were measured on. vLLM 0.24.0 logs 'Using TRITON Fp8 MoE backend out of potential backends: [AITER, FLASHINFER_TRTLLM, FLASHINFER_CUTLASS, VLLM_CUTLASS, TRITON, MARLIN, BATCHED_VLLM_CUTLASS, BATCHED_TRITON, XPU, CPU]'. Do not override it with --moe-backend on the strength of advice written for a different precision: the unquantized build of this same model picks FlashInfer CUTLASS instead, and neither choice transfers.

'To serve at least one request with the model's max seq len (262144), 5.xx GiB KV cache is needed, which is larger than the available KV cache memory (4.0 GiB).'

Your --kv-cache-memory-bytes is too small. The intuition that a smaller checkpoint should need a smaller KV pin is wrong: this quantization does not touch the cache (kv_cache_scheme is null) and per-token KV is a function of the architecture, so the full 262,144-token window costs the same little over 5 GiB here as it does in BF16. Use 8589934592 (8 GiB), which yields 412,980 tokens and 1.58x concurrency. Do not solve it by lowering --max-model-len; there is over 60 GiB free on this config, so giving up a quarter of the context would buy nothing.

The log says 'Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size'.

Expected on this architecture, not a warning to act on. Thirty of the forty language layers are gated-delta linear attention carrying a per-sequence recurrent state, whose page has to fit alongside the full-attention pages — which forces the unusually large block size. It is part of why per-token KV lands where it does.

Engine core initialization failed with a list of uninitialized visual.* parameters.

You are not serving this checkpoint. Several models register the same Qwen3_5MoeForConditionalGeneration architecture as language-only releases that still carry a vision_config, and the class builds its 27-block vision tower unconditionally — so those fail on ~113 uninitialized visual.* weights and need both --language-model-only and a source patch to vllm/model_executor/models/qwen3_5.py. This FP8 export is not affected: its index contains all 333 visual.* tensors, left in BF16 by the quantization. If you see this error, check `python3 -c "import json;print(sum('visual' in k for k in json.load(open('model.safetensors.index.json'))['weight_map']))"` on the tree you are actually pointing at.

Can I turn on speculative decoding? The config says mtp_num_hidden_layers: 1.

No draft ships with this checkpoint. The config advertises an MTP layer but model.safetensors.index.json holds 62,546 tensors and not one of them matches 'mtp' — the language layers run 0-39 with nothing at index 40, where a native MTP layer would sit. Passing '{"method":"mtp"}' will not find weights to load. Always grep the index, not just the config; the config is a declaration of intent and the index is the fact.

How much of the speedup over the unquantized build is real, and where does it come from?

Measured on the same node with identical flags, warm, greedy, thinking off, at a 2048-token prompt: decode 30.25 tok/s unquantized versus 37.70 here (+24.6%), prefill 3,746 versus 5,096 tok/s (+36%). The decode gain is a bandwidth effect — the routed experts are 92% of the parameters and this quantization compresses them, so a token reads roughly half as many bytes. It is not the full 2x because the 30 gated-delta linear-attention layers, the router gates, the embeddings and lm_head are all on the ignore list and stay BF16. The larger prefill gain is a compute effect: prefill is arithmetic-bound and FP8 tensor cores do that arithmetic faster. If a vendor quant's ignore list had covered only the experts and left full attention in BF16, the decode number would move far less — check the list before predicting the gain.

A 200K-token prompt takes minutes before the first token appears.

Expected, and it is the number that decides whether this model is usable on a large repository. Measured TTFT on one Spark: 0.19 s at 511 tokens, 0.40 s at 2,042, 1.35 s at 8,208, 6.0 s at 32,780, 38.3 s at 131,292 and 105.7 s at 250,223. Decode over the same sweep falls from 37.9 to 21.0 tok/s, so the long-context cost is almost entirely prefill. Two things worth knowing: the quantization helps prefill at length less than it helps a short prompt (2,367 tok/s at 250K against 6,062 at 8K, because attention over the growing prefix comes to dominate), and the KV cache is not quantized at all, so the decode advantage over an unquantized serve also shrinks as the prompt grows. Raising --max-num-batched-tokens above 8192 trades startup transients for a larger prefill chunk if you want to push on it.

I have 60 GiB free while serving. What should I spend it on?

Not more KV — the pool is already at 1.58x concurrency for a single user, and multiples above 1.0x are context you paid for and cannot use. The honest answers are: a resident speculative draft if one is ever published for this model (none exists today), a second service on the same box, or nothing at all. Headroom on a unified-memory machine is not waste; an overcommit here does not OOM the process, it swap-wedges the whole node.

Should I serve this on two Sparks with TP2 instead?

There is no reason to. The whole working set is ~51 of 114 usable GiB on one node. Tensor parallelism would buy memory that is not scarce while adding the RoCE fabric setup, the per-node NCCL_IB_GID_INDEX resolution, the Gloo interface binding and the cross-node CUDA-graph deadlock that forces --enforce-eager. Keep it single-node — and use the second Spark to sweep flags in parallel instead.