BTL-3 (Qwen3.6-27B) — BF16

badtheorylabs' BTL-3 agentic coder (a LoRA on Qwen3.6-27B, ~27B dense) in BF16 on one DGX Spark, with the base model's native MTP head driving speculative decode — 12.8 tok/s single-stream, 2.9x the no-draft baseline, at the full 262K context.

BTL-3 (badtheorylabs) is an agentic-coding / tool-use model, and it is not a standalone model at all -- it is a rank-32 PEFT LoRA adapter on Qwen/Qwen3.6-27B. Qwen3.6-27B is a dense, hybrid-attention (mixed full + linear/gated-delta) 27B in the qwen3_5 family, with a small vision tower and a native MTP head. This recipe is the vanilla way to run BTL-3 on one Spark: MERGE the LoRA into the base once (peft merge_and_unload -> a plain BF16 checkpoint), then serve it with vLLM using the base's own MTP layer as a speculative drafter. Merging avoids any vLLM LoRA-support risk on the linear-attention mixer layers the adapter targets and lets the native draft load cleanly. The whole story is decode speed: a dense 27B in BF16 is memory-bandwidth bound (~54 GB read per token) and decodes at only ~4.4 tok/s on GB10's ~273 GB/s. Speculative decode off the native MTP head is the lever -- swept to its peak at k=4 it reaches 12.8 tok/s single-stream (2.9x), because BTL-3's coding output is highly predictable so draft acceptance stays high several tokens deep. Context is cheap on this hybrid arch, so the full 262K window fits with room to spare; for one user we cap KV and hand the rest back.

• 1 x DGX Spark (GB10, sm_121), single node -- the full ~27B model, BF16, no quant
• BTL-3 = a rank-32 LoRA on Qwen3.6-27B; served MERGED (peft merge_and_unload) as a plain BF16 checkpoint
• The base's NATIVE MTP head drives speculative decode -- but the LoRA merge drops it, so its 15 mtp.* tensors are grafted back from the base checkpoint
• MTP k swept to its peak: k=4 gives 12.82 tok/s vs 4.4 no-draft (2.9x); k=5 regresses to 12.66
• Measured single-stream decode 12.81 tok/s at a 2048 prompt; nearly flat across length (12.7 / 12.8 / 12.3 at 512 / 2048 / 8192); BF16 is bandwidth-bound (~4.4 tok/s floor)
• Hybrid attention (16 of 64 layers full) -> KV is ~64.7 KiB/token; the full 262K context fits, capped to ~1.2x for one user
• vLLM 0.25.1 serves the qwen3_5 arch on sm_121 out of the box -- no engine patch; the only trap is ninja/nvcc on PATH for the flashinfer sampler JIT
• Vision tower disabled for text-only serving (--limit-mm-per-prompt image:0,video:0)

Software requirements

• 1 x DGX Spark (GB10, sm_121), NVIDIA driver >= 580 (CUDA-13 capable)
• vLLM 0.25.1 + torch 2.11.0+cu130 in a venv (serves qwen3_5 / Qwen3_5ForConditionalGeneration on sm_121 with no patch)
• peft 0.19.1 + transformers 5.10.1 for the one-time LoRA merge (pip install --no-deps peft into the serving venv is enough)
• ninja + nvcc on the serving process PATH -- vLLM's flashinfer top-k/top-p sampler JIT-compiles a kernel on first use
• ~51 GB free disk for the merged checkpoint

Quick start

1. 1

   Download the base + adapter

   BTL-3 is a LoRA adapter (890 MB) plus a tokenizer and an agentic chat template; the 27B of weights all live in the base Qwen/Qwen3.6-27B (~52 GB, 15 shards). You need both to merge.

   hf download Qwen/Qwen3.6-27B --local-dir ~/models/hf/Qwen3.6-27B
   hf download badtheorylabs/BTL-3 --local-dir ~/models/hf/BTL-3bash

2. 2

   Merge the LoRA into the base (one time, CPU, bf16)

   Serve BTL-3 as a plain checkpoint by folding the adapter into the base with peft. The adapter targets attention + MLP + linear-attn mixer projections via a regex; merge_and_unload applies them all. Copy BTL-3's tokenizer + chat_template over the base's (that is BTL-3's tool-use template) and keep the base's vision preprocessor configs. Note transformers' Qwen3_5ForConditionalGeneration does NOT instantiate the MTP head, so the saved checkpoint is missing the base's 15 mtp.* tensors -- the next step grafts them back.

   import os, shutil, torch
   from transformers import AutoModelForImageTextToText
   from peft import PeftModel
   BASE=os.path.expanduser('~/models/hf/Qwen3.6-27B'); ADAPTER=os.path.expanduser('~/models/hf/BTL-3')
   OUT=os.path.expanduser('~/models/hf/BTL-3-merged')
   m=AutoModelForImageTextToText.from_pretrained(BASE, dtype=torch.bfloat16, low_cpu_mem_usage=True)
   m=PeftModel.from_pretrained(m, ADAPTER)
   m=m.merge_and_unload()
   os.makedirs(OUT, exist_ok=True)
   m.save_pretrained(OUT, safe_serialization=True, max_shard_size='5GB')
   for f in os.listdir(BASE):                       # base non-weight files (vision preprocessor, generation_config)
       s=os.path.join(BASE,f)
       if os.path.isdir(s) or f.endswith('.safetensors') or f=='model.safetensors.index.json': continue
       if not os.path.exists(os.path.join(OUT,f)): shutil.copy2(s, os.path.join(OUT,f))
   for f in ['tokenizer.json','tokenizer_config.json','chat_template.jinja','processor_config.json']:  # BTL-3's
       if os.path.exists(os.path.join(ADAPTER,f)): shutil.copy2(os.path.join(ADAPTER,f), os.path.join(OUT,f))python

   Skip the base's .cache/ directory in the copy loop or shutil.copy2 raises IsADirectoryError. Result: ~/models/hf/BTL-3-merged, 12 shards, ~51 GB, config.json still says mtp_num_hidden_layers: 1.

3. 3

   Graft the native MTP tensors back into the merged checkpoint

   The merge dropped the base's MTP layer (transformers' CausalLM class never built it), so config.json claims an MTP head with no weights -- and vLLM's drafter then dies with 'Following weights were not initialized from checkpoint: {model.norm.weight, model.pre_fc_norm_embedding.weight, ...}'. The 15 mtp.* tensors (849 MB) live in the base checkpoint and the LoRA regex never touches them, so copy them verbatim into the merged dir and add them to its index.

   import os, json
   from safetensors import safe_open
   from safetensors.torch import save_file
   BASE=os.path.expanduser('~/models/hf/Qwen3.6-27B'); OUT=os.path.expanduser('~/models/hf/BTL-3-merged')
   bwm=json.load(open(f'{BASE}/model.safetensors.index.json'))['weight_map']
   keys=[k for k in bwm if k.startswith('mtp.')]
   by_shard={}
   for k in keys: by_shard.setdefault(bwm[k], []).append(k)
   t={}
   for shard,ks in by_shard.items():
       with safe_open(f'{BASE}/{shard}', framework='pt') as f:
           for k in ks: t[k]=f.get_tensor(k)
   save_file(t, f'{OUT}/model-mtp.safetensors', metadata={'format':'pt'})
   sz=os.path.getsize(f'{OUT}/model-mtp.safetensors')
   idx=json.load(open(f'{OUT}/model.safetensors.index.json'))
   for k in keys: idx['weight_map'][k]='model-mtp.safetensors'
   idx.setdefault('metadata',{})['total_size']=idx['metadata'].get('total_size',0)+sz
   json.dump(idx, open(f'{OUT}/model.safetensors.index.json','w'), indent=2)python

   After this the merged checkpoint has 1199 tensors (matching the base) and the drafter loads. If you do NOT want speculative decode, you can skip this and serve without --speculative-config -- but you cap decode at ~4.4 tok/s.

4. 4

   Serve on one Spark with MTP k=4

   vLLM 0.25.1 serves the qwen3_5 arch on sm_121 with no patch. Three things matter: (1) ninja + nvcc must be on the process PATH -- vLLM's flashinfer sampler JIT-compiles a kernel on the first sampling call and dies with FileNotFoundError: 'ninja' otherwise; (2) disable the vision tower for text-only; (3) cap KV for a single user (the pool is 2.99x uncapped -- context you cannot use). k=4 is the swept peak of the native MTP draft.

   export PATH="$HOME/venvs/vllm-025/bin:/usr/local/cuda/bin:$HOME/.local/bin:$PATH"  # ninja + nvcc
   
   vllm serve ~/models/hf/BTL-3-merged \
     --served-model-name btl3 \
     --chat-template ~/models/hf/BTL-3-merged/chat_template.jinja \
     --limit-mm-per-prompt '{"image":0,"video":0}' \
     --max-model-len 262144 \
     --kv-cache-memory-bytes 21474836480 \
     --gpu-memory-utilization 0.85 \
     --max-num-seqs 4 --max-num-batched-tokens 8192 \
     --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
     --host 0.0.0.0 --port 8000bash

   First boot loads 13 shards (~5 min cold) then JITs the sampler kernel once. Expect 'Loading drafter model...' with no error and, after traffic, 'SpecDecoding metrics: ... Per-position acceptance rate: ...' in the log.

5. 5

   Smoke test (thinking off) and confirm the drafter accepts

   Confirm coherence and that the MTP draft is actually accepting -- a broken draft is silent (verification just rejects it) and shows up only as decode sinking toward the 4.4 tok/s baseline.

   curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
     -d '{"model":"btl3","messages":[{"role":"user","content":"Write a Python function to reverse a linked list."}],"max_tokens":128,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}'
   
   # acceptance from the log (or /metrics):
   grep 'Per-position acceptance' ~/serve.log | tail -1
   # k=4 measured: 0.938 / 0.719 / 0.594 / 0.438, mean accepted length 3.69bash

   BTL-3 is a thinking model -- enable_thinking is a per-request chat_template_kwargs flag, not a serve flag. Turn it OFF for throughput benchmarking.

Key vLLM parameters

API usage

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"btl3",
       "messages":[{"role":"user","content":"Write a production-quality LRU cache in Python with get and put."}],
       "max_tokens":256, "temperature":0, "stream":true,
       "stream_options":{"include_usage":true},
       "chat_template_kwargs":{"enable_thinking":false}}'
# decode_tps = (completion_tokens - 1) / (t_last_token - t_first_token)bash

Troubleshooting

Drafter fails at load: 'Following weights were not initialized from checkpoint: {model.norm.weight, model.pre_fc_norm_embedding.weight, model.pre_fc_norm_hidden.weight, model.layers.0.self_attn.q_norm.weight, ...}'

The LoRA merge dropped the base's native MTP head -- transformers' Qwen3_5ForConditionalGeneration never instantiates it, so save_pretrained wrote a checkpoint whose config still says mtp_num_hidden_layers: 1 but which has no mtp.* tensors. Graft the 15 mtp.* tensors from the base Qwen3.6-27B checkpoint into the merged dir and add them to the index (see the graft step). The LoRA regex never touches mtp.* so they copy verbatim.

FileNotFoundError: [Errno 2] No such file or directory: 'ninja' -- engine dies right after 'Initial profiling/warmup run took'

vLLM's flashinfer top-k/top-p sampler (topk_topp_sampler.flashinfer_sample -> flashinfer/jit/cpp_ext.py run_ninja) JIT-compiles a kernel on the first sampling call. A detached/non-login serve shell lacks ninja (in ~/venvs/vllm-025/bin) and nvcc (/usr/local/cuda/bin) on PATH. Export both before serving.

Decode quietly sits near 4.4 tok/s with MTP enabled

The draft is being rejected (a broken draft is silent -- verification just discards it). Check 'Per-position acceptance rate' in the log or spec_decode_num_accepted_tokens_total in /metrics; it should be well above 0. On BTL-3 coding prompts k=4 accepts 0.94/0.72/0.59/0.44.

Cold start takes ~5-6 minutes and reloads weights every restart

The 12 shards load cold from NVMe (EXT4, ~28 s/shard) when the page cache has been evicted -- a large uncapped KV pool (48 GiB) pushes the weights out. Capping KV (as the published config does) leaves the weights resident in page cache, so restarts reload fast.

Which k to run?

Swept at a 2048 prompt, warm, thinking off: k=1 7.66, k=2 10.09, k=3 11.67, k=4 12.82, k=5 12.66 tok/s. k=4 is the peak; k=5 regresses because the 5th draft position's acceptance (~0.27) no longer offsets its compute. Mean accepted length plateaus at ~3.7 past k=4.