Ornith 1.0 9B — BF16

Deep Reinforce's agentic-coding Ornith 1.0 9B (9B dense) unquantized in BF16 on one DGX Spark, serving the full 262,144-token context at 12.6 tok/s single-stream.

Contributors

@kvncrw

Ornith 1.0 9B is the small member of Deep Reinforce's agentic-coding family, and unlike its 35B and 397B siblings it is dense: 32 layers at hidden_size 4096, an MLP intermediate of 12288, a 262,144-token native window, and a 27-block vision tower whose weights are all present in the checkpoint. At 17.5 GiB of BF16 it is the easiest thing on this hardware to place — the weights are 15% of a Spark, the whole serving working set is about 39 of 114 usable GiB, and there is no quantization artifact to audit, no calibration set to defend, and no second node to coordinate. The reason to read this page carefully is the number it publishes: 12.6 tok/s single-stream at a 2048-token prompt. That is slow, and it is slow for a structural reason worth understanding before you pick this model. Decode on a Spark is bandwidth-bound, so felt speed tracks the bytes one token reads, and a dense model reads all of them. Summed from the safetensors headers, a decode step here pulls 15.87 GB — everything except the vision tower and the embedding gather — which against the Spark's published 273 GB/s of LPDDR5X puts the arithmetic ceiling at 17.2 tok/s. The measured 12.6 is 73% of that, which is a healthy fraction and means nothing is misconfigured; it means the model is asking for a lot of bandwidth. Total parameter count is a poor guide here, and the direction of the error surprises people: a sparse mixture-of-experts several times larger can be several times faster, because it reads only the experts it routes to. There is no speculative draft to recover the difference. The config advertises mtp_num_hidden_layers: 1, but of the 760 tensors in the weight index not one matches 'mtp' and the layers stop at index 31, where a native MTP layer would sit at 32. So take this recipe for what it is good at. It loads in under two minutes, leaves three quarters of the machine free, serves the full 262,144-token window with a proven request at the top of it, and gives you the model exactly as it was published. If you want speed on this hardware, the lever is fewer bytes per token — a 4-bit build of these weights, or a sparse model — not fewer parameters.

• 1 x DGX Spark (GB10, sm_121) — no tensor parallelism, no second node
• Native BF16, exactly as Deep Reinforce published it — no quantization, no calibration, nothing to audit
• Measured 12.6 tok/s single-stream decode at a 2048-token prompt, warm, greedy
• Dense, so every token reads 15.87 GB of weights — 73% of the 17.2 tok/s arithmetic ceiling at the Spark's published 273 GB/s
• Full 262,144-token context served, at 1.49x concurrency on a 12 GiB pinned KV pool
• Working set ~39 of 114 usable GiB — about three quarters of the machine stays free while serving the full window
• Loads on stock vLLM 0.24.0 with no patches and no --language-model-only: all 333 vision-tower tensors are present
• Hybrid attention: 24 of 32 layers are gated-delta linear, 8 are full attention — but those 8 keep 4 KV heads, so KV costs 32 KiB/token
• No speculative draft available — the config advertises mtp_num_hidden_layers: 1 but the checkpoint ships zero MTP tensors
• Decode falls gently with prompt length (12.65 / 12.61 / 12.50 / 12.00 / 10.34 / 8.86 tok/s at 0.5K / 2K / 8K / 32K / 131K / 250K) while TTFT climbs from 0.14 s to 109 s
• Fast to bring up: 108 s of weight loading from cold NVMe, then 37 s of profiling, compile and warmup

Software requirements

• 1 x DGX Spark (GB10, sm_121), NVIDIA driver >= 580 (CUDA-13 capable)
• vLLM 0.24.0 with torch 2.11.0+cu130 (verified 2026-07-27). The architecture Qwen3_5ForConditionalGeneration is registered in vLLM 0.24.0
• The `hf` CLI to fetch the checkpoint (~18.8 GB)
• ~25 GB of free disk on the serving node

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ornith9b",
    "messages": [{"role": "user", "content": "Explain gated-delta linear attention in three sentences."}],
    "max_tokens": 512,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | jq -r '.choices[0].message.content'bash

import json, time, urllib.request, uuid

body = {
    "model": "ornith9b",
    "messages": [{"role": "user",
                  "content": f"[{uuid.uuid4().hex[:8]}] Write an LRU cache in Python."}],
    "max_tokens": 256, "temperature": 0, "stream": True,
    "stream_options": {"include_usage": True}, "ignore_eos": True,
    "chat_template_kwargs": {"enable_thinking": False},
}
req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions",
                             data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
t0 = time.perf_counter(); t_first = None; usage = None
with urllib.request.urlopen(req) as res:
    for raw in res:
        line = raw.decode().strip()
        if not line.startswith("data: "):
            continue
        if line[6:] == "[DONE]":
            break
        ch = json.loads(line[6:])
        usage = ch.get("usage") or usage
        if ch.get("choices") and ch["choices"][0]["delta"].get("content") and not t_first:
            t_first = time.perf_counter()
t_end = time.perf_counter()
print(f"ttft {1000*(t_first-t0):.0f} ms")
print(f"decode {(usage['completion_tokens']-1)/(t_end-t_first):.1f} tok/s")
# expect ~12.6 tok/s warm at a short promptpython

import json, glob, struct, os

tot = {}
for f in sorted(glob.glob(os.path.expanduser(
        "~/models/hf/Ornith-1.0-9B/*.safetensors"))):
    with open(f, "rb") as fh:
        n = struct.unpack("<Q", fh.read(8))[0]
        header = json.loads(fh.read(n))
    for k, v in header.items():
        if k == "__metadata__":
            continue
        lo, hi = v["data_offsets"]
        if ".visual." in k or k.startswith("model.visual"):
            c = "vision_tower"          # only runs on image inputs
        elif "embed_tokens" in k:
            c = "embed_tokens"          # a gather, not a full read
        else:
            c = "read_every_token"
        tot[c] = tot.get(c, 0) + hi - lo

read = tot["read_every_token"]
print(f"bytes read per decode token: {read / 1e9:.2f} GB")
print(f"ceiling at 273 GB/s:         {273 / (read / 1e9):.1f} tok/s")
# bytes read per decode token: 15.87 GB
# ceiling at 273 GB/s:         17.2 tok/spython

grep -E 'GPU KV cache size|Maximum concurrency' /tmp/vllm-ornith9b.log
# GPU KV cache size: 390,070 tokens
# Maximum concurrency for 262,144 tokens per request: 1.49xbash

Troubleshooting

Decode is only ~12.6 tok/s. A 9B model should be fast — something must be misconfigured.

Nothing is misconfigured; this is the model asking for bandwidth. Decode on a Spark is bandwidth-bound, and a dense model reads every weight on every token. Summed from the safetensors headers, a decode step here pulls 15.87 GB — the dense MLP is 9.00 GiB of it, the gated-delta linear-attention projections 3.01, the lm_head 1.90, the full-attention projections 0.88. Against the Spark's published 273 GB/s that is a 17.2 tok/s arithmetic ceiling, and the measured 12.6 is 73% of it, which is a normal fraction. Before blaming a kernel, run the byte-accounting snippet above: if your measurement is a healthy fraction of the ceiling, the only lever left is fewer bytes per token — a 4-bit build of these weights, or a sparse model that routes to a fraction of its parameters. Parameter count is the wrong number to reason from.

'To serve at least one request with the model's max seq len (262144), 8.xx GiB KV cache is needed, which is larger than the available KV cache memory'.

Your --kv-cache-memory-bytes is too small. This trips people because the model is small and the intuition is that its cache must be too — it is not. Eight of the thirty-two layers are full attention with 4 KV heads at head_dim 256, so a token costs 32 KiB and the full window costs just over 8 GiB. Use 12884901888 (12 GiB), which yields 390,070 tokens and 1.49x concurrency. Do not solve it by lowering --max-model-len: the message helpfully suggests a smaller window, but memory is nowhere near scarce on this node — three quarters of it is free — so giving up context to save cache is a straight loss.

Every answer starts by explaining its plan before writing any code.

That is the model reasoning, not a template bug. Ornith's chat_template.jinja opens the assistant turn with '<think>\n' unless the request passes enable_thinking: false, in which case it emits an empty '<think>\n\n</think>\n\n' pair and the answer starts immediately. Send it as "chat_template_kwargs": {"enable_thinking": false}. Decide this before you benchmark: with thinking on, most of the generated tokens are reasoning, so the throughput numbers are not comparable with the ones here.

'ValueError: There is no module or parameter named visual in Qwen3_5ForConditionalGeneration' at weight load.

You passed --language-model-only, or you are on a vLLM venv that carries a vision-tower guard patch for a different checkpoint on this architecture family. This checkpoint ships the tower — 333 visual.* tensors, 0.85 GiB — so the tower must be built. Drop the flag. The mirror-image failure, a pile of uninitialized visual.* parameters, is what you get when a language-only checkpoint on this family is served without the flag; the weight index is the only thing that settles which case you are in. Check it with: python3 -c "import json;w=json.load(open('model.safetensors.index.json'))['weight_map'];print(sum('visual' in k for k in w))" — greater than zero means no flag.

The log says 'Setting attention block size to N tokens to ensure that attention page size is >= mamba page size'.

Expected on this architecture, not a warning to act on. Twenty-four of the thirty-two layers are gated-delta linear attention carrying a per-sequence recurrent state, whose page has to fit alongside the full-attention pages — which forces an unusually large block size. It is part of why per-token KV lands where it does.

'Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre)'.

Informational. vLLM only prefetches whole trees on network filesystems; on local NVMe it reads shards as it needs them. Loading took 108 s cold and there is nothing to tune here. Note also that you should NOT purge page cache before this serve: --kv-cache-memory-bytes skips vLLM's startup free-memory check entirely, so the purge buys nothing and only costs you a cold read.

A 250K-token prompt seems to hang.

It is prefilling. Measured TTFT at a 250,223-token prompt is 109 seconds, and prefill throughput has fallen to 2,303 tok/s by then from a peak of 4,430 at 8K. The context is real — a single 261,224-token request served and then decoded — but the first token on a very long prompt costs minutes, and that, not decode, is what decides whether this model is pleasant to use on a large repository. Raise your client timeout rather than shortening the window.

Ruled out: enabling speculative decoding to recover the decode speed.

There is no draft to enable. The config advertises mtp_num_hidden_layers: 1, but the weight index contains zero tensors matching 'mtp' and the language layers stop at index 31, so a {"method":"mtp"} speculative config has nothing to load. An ngram/prompt-lookup draft would at least initialize on this architecture — vLLM's gated-delta linear-attention path partitions the recurrent state for spec decode properly, so it does not hit the assertion that a bare linear-attention port would — but prompt-lookup only drafts when the output repeats the prompt, which is a net loss on the open-ended generation this benchmark measures. It was not measured here.