Ornith 1.0 35B — BF16

Deep Reinforce's agentic-coding Ornith 1.0 35B (35B MoE, ~3B active) unquantized in BF16 on one DGX Spark, serving the full 262,144-token context at 30.3 tok/s single-stream.

Contributors

@kvncrw

Ornith 1.0 35B is Deep Reinforce's agentic-coding model, post-trained on the Qwen3.5 35B-A3B MoE stack: 256 routed experts at top-8, roughly 3B active parameters per token, 30 of its 40 layers gated-delta linear attention and 10 full attention, a 262,144-token native window, and MIT licensed. At 70.2 GB of BF16 it lands on a single Spark with room to spare, which is the reason to run it unquantized — you get the model exactly as it was published, with no quantization artifact to audit and no calibration set to defend. It is a genuinely multimodal release: the checkpoint carries all 333 tensors of a 27-block vision tower alongside the language model, so it loads on stock vLLM with no patching, which is not true of every checkpoint that declares this architecture. The price of BF16 is decode speed. Every generated token re-reads roughly 3B active parameters at two bytes each, and decode on the Spark is bandwidth-bound, so the measured 30.3 tok/s single-stream is what the memory bus allows; a 4-bit expert build is the lever if you want it faster, and this recipe is the honest BF16 reference such a build would be measured against. Two things are worth knowing before you copy the serve command. First, there is no free speculative draft: the config advertises mtp_num_hidden_layers: 1, but model.safetensors.index.json contains 31,666 tensors and not one of them matches 'mtp' — the language layers run 0 to 39 with nothing at index 40, where a native MTP layer would sit. Second, memory is not the binding constraint but it is not free either. The weights alone take 65.53 GiB of a 114 GiB budget, so unlike a 4-bit build of the same architecture you do not get to be careless with the KV pool: an 8 GiB pin gives 1.58x concurrency at the full context, and a 4 GiB pin — which is ample for a 4-bit sibling — cannot fit a single full-length request. Long prompts are served but they are not fast to start: the largest request the window admits, 261,224 prompt tokens, took 122 seconds before its first token and then decoded at 18.1 tok/s. The context claim here is measured rather than inherited — that request is in the committed benchmark rows — and it is the TTFT curve, not decode, that decides whether this model is pleasant to use on a large repository.

• 1 x DGX Spark (GB10, sm_121) — no tensor parallelism, no second node
• Native BF16, exactly as Deep Reinforce published it — no quantization, no calibration, nothing to audit
• Measured 30.3 tok/s single-stream decode at a 2048-token prompt, warm, greedy
• Full 262,144-token context served, at 1.58x concurrency on an 8 GiB pinned KV pool
• Loads on stock vLLM 0.24.0 with no patches and no --language-model-only: the vision tower's weights are all present
• Hybrid attention: 30 of 40 layers are gated-delta linear, 10 are full attention — which is why a 262K window costs only 8 GiB of pinned KV
• No speculative draft available — the config advertises mtp_num_hidden_layers: 1 but the checkpoint ships zero MTP tensors
• Decode falls gently with prompt length (30.4 / 30.3 / 29.8 / 28.0 / 22.5 / 18.3 tok/s at 0.5K / 2K / 8K / 32K / 131K / 250K) while TTFT climbs from 0.3 s to 114 s
• The claimed context is proven, not assumed: a single 261,224-token prompt served at 18.1 tok/s decode after 122 s of prefill
• Working set ~82 of 114 usable GiB; weights alone are 65.53 GiB
• Weight load ~440 s cold from NVMe, then ~78 s of profiling, compile and warmup

Software requirements

• 1 x DGX Spark (GB10, sm_121), NVIDIA driver >= 580 (CUDA-13 capable)
• vLLM 0.24.0 with torch 2.11.0+cu130 (verified 2026-07-27). The architecture Qwen3_5MoeForConditionalGeneration is also registered in vLLM 0.25.1 and 0.26.0
• The `hf` CLI to fetch the checkpoint (~70.3 GB)
• ~75 GB of free disk on the serving node

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ornith",
    "messages": [{"role": "user", "content": "Explain MoE routing in three sentences."}],
    "max_tokens": 512,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | jq -r '.choices[0].message.content'bash

import json, time, urllib.request, uuid

body = {
    "model": "ornith",
    "messages": [{"role": "user",
                  "content": f"[{uuid.uuid4().hex[:8]}] Write an LRU cache in Python."}],
    "max_tokens": 256, "temperature": 0, "stream": True,
    "stream_options": {"include_usage": True}, "ignore_eos": True,
    "chat_template_kwargs": {"enable_thinking": False},
}
req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions",
                             data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
t0 = time.perf_counter(); t_first = None; usage = None
with urllib.request.urlopen(req) as res:
    for raw in res:
        line = raw.decode().strip()
        if not line.startswith("data: "):
            continue
        if line[6:] == "[DONE]":
            break
        ch = json.loads(line[6:])
        usage = ch.get("usage") or usage
        if ch.get("choices") and ch["choices"][0]["delta"].get("content") and not t_first:
            t_first = time.perf_counter()
t_end = time.perf_counter()
print(f"ttft {1000*(t_first-t0):.0f} ms")
print(f"decode {(usage['completion_tokens']-1)/(t_end-t_first):.1f} tok/s")python

grep -E 'GPU KV cache size|Maximum concurrency' /tmp/vllm-ornith.log
# GPU KV cache size: 412,980 tokens
# Maximum concurrency for 262,144 tokens per request: 1.58xbash

Troubleshooting

Every answer starts with 'Here's a thinking process: 1. Understand User Request...' before any code.

That is the model reasoning, not a template bug. Ornith's chat_template.jinja opens the assistant turn with '<think>\n' unless the request passes enable_thinking: false, in which case it emits an empty '<think>\n\n</think>\n\n' pair and the answer starts immediately. Send it as "chat_template_kwargs": {"enable_thinking": false}. Decide this before you benchmark: with thinking on, most of the generated tokens are reasoning, so the throughput numbers are not comparable with the ones here.

'To serve at least one request with the model's max seq len (262144), 5.xx GiB KV cache is needed, which is larger than the available KV cache memory (4.0 GiB).'

Your --kv-cache-memory-bytes is too small for BF16. A 4 GiB pool is enough for a 4-bit build of this architecture but not for this one. Use 8589934592 (8 GiB), which yields 412,980 tokens and 1.58x concurrency. Do not solve it by lowering --max-model-len; the message helpfully suggests a smaller window, but memory is not scarce enough on this node to give up a quarter of the context for it.

The log says 'Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size'.

Expected on this architecture, not a warning to act on. Thirty of the forty language layers are gated-delta linear attention carrying a per-sequence recurrent state, whose page has to fit alongside the full-attention pages — which forces the unusually large block size. It is part of why per-token KV lands where it does.

'Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (65.40 GiB) exceeds 90% of available RAM'.

Informational. It means vLLM will not try to prefetch the whole tree into page cache before loading it, which is correct here — the checkpoint is over half the machine's memory. It is also why the first load takes 440 s: the shards are read from NVMe as they are needed. A second boot with the tree still in page cache is much faster. Do not attempt to fix this by purging the cache; --kv-cache-memory-bytes already skips the free-memory check that a purge exists to satisfy, so a purge only costs you a full re-read.

Decode is only ~30 tok/s. Is a kernel misconfigured?

No — that is what BF16 costs on this hardware. Decode on the Spark is bandwidth-bound and each generated token re-reads roughly 3B active parameters at two bytes each. The measured 30.3 tok/s at a 2048-token prompt is a normal fraction of the resulting roofline, and the backends chosen (FlashInfer CUTLASS unquantized MoE, FLASH_ATTN) are the right ones. If you want this model faster, the lever is a 4-bit expert quantization, not a flag — and be aware that on a 92%-expert MoE like this one a quant that leaves attention unquantized buys much less decode than its file-size reduction suggests, because unquantized attention can still dominate per-token bytes read.

Can I turn on speculative decoding? The config says mtp_num_hidden_layers: 1.

No draft ships with this checkpoint. The config advertises an MTP layer but model.safetensors.index.json holds 31,666 tensors and not one of them matches 'mtp' — the language layers run 0-39 with nothing at index 40, where a native MTP layer would sit. Passing '{"method":"mtp"}' will not find weights to load. Always grep the index, not just the config; the config is a declaration of intent and the index is the fact.

A 200K-token prompt takes minutes before the first token appears.

Expected, and it is the number that decides whether this model is usable on a large repository. Measured TTFT on one Spark: 0.31 s at 511 tokens, 0.54 s at 2,042, 1.65 s at 8,206, 7.1 s at 32,778, 42.9 s at 131,290, 114.0 s at 250,223 and 122.0 s at 261,224 — the largest prompt the window admits, which does serve, at 18.1 tok/s decode. Decode barely moves over the same sweep (30.4 down to 18.1 tok/s), so the long-context cost is almost entirely prefill. Raising --max-num-batched-tokens above 8192 trades startup transients for a larger prefill chunk if you want to push on it; the ceiling is the ~4,975 tok/s prefill rate this config reaches around an 8K prompt.

Engine core initialization failed with a list of uninitialized visual.* parameters.

You are not serving this checkpoint. Several models register the same Qwen3_5MoeForConditionalGeneration architecture as language-only releases that still carry a vision_config, and the class builds its 27-block vision tower unconditionally — so those fail on ~113 uninitialized visual.* weights and need both --language-model-only and a source patch to vllm/model_executor/models/qwen3_5.py. Ornith 1.0 35B is not affected: its index contains all 333 visual.* tensors. If you see this error, check `python3 -c "import json;print(sum('visual' in k for k in json.load(open('model.safetensors.index.json'))['weight_map']))"` on the tree you are actually pointing at.

Should I serve this on two Sparks with TP2 instead?

There is no reason to. The whole working set is ~82 of 114 usable GiB on one node, so tensor parallelism would buy memory that is not scarce while adding the RoCE fabric setup, the per-node NCCL_IB_GID_INDEX resolution, the Gloo interface binding and the cross-node CUDA-graph deadlock that forces --enforce-eager. Keep it single-node — and use the second Spark to sweep flags in parallel instead.