AREX-Base 122B-A10B — Q4_K_M

BAAI's AREX-Base deep-research agent (122B MoE, ~10B active) in Q4_K_M on one DGX Spark — 23.98 tok/s single-stream, at a measured 250,223-token context.

Contributors

@Marinara_Waif

AREX-Base is BAAI's deep-research agent built on the Qwen3.5-122B-A10B stack — a 256-expert MoE routing top-8, with 36 of its 48 layers running gated-delta linear attention and only every fourth layer keeping full attention. At BF16 it is 245.1 GB and needs more memory than two Sparks have; at Q4_K_M it is 74.9 GB and needs one, which is the whole reason this recipe is a llama.cpp recipe rather than a vLLM one. No FP8 or NVFP4 build of this checkpoint exists on the Hub, and quantizing 245 GB ourselves is a day's work, so the fastest honest route to a running 122B model on a single Spark is the community GGUF. What you get is a model that decodes at 23.98 tok/s single-stream at a 2,048-token prompt and holds that rate over any prompt length you would call ordinary — 23.99 at 512 tokens, 23.98 at 2,048, 23.66 at 8,192 — because 36 of 48 layers have no growing cache to re-read. It is not flat forever: decode falls to 17.40 tok/s at a 131,292-token prompt and 13.91 at 250,223, i.e. 42% off the short-prompt rate at the top of the window, which is the twelve full-attention layers finally making themselves felt. The short-prompt number is a healthy fraction of what the hardware allows: roughly 10.2B parameters are active per token (6.6B of always-read layers plus 8/256 of the 115.9B expert plane), which at the file's average 4.9 bits per parameter is about 6.2 GB read per token, so GB10's 273 GB/s caps this configuration near 44 tok/s and the measurement sits at 54% of that. The real cost of a long prompt is not decode, it is prefill: at 483–704 tok/s it barely improves with length, so time-to-first-token runs 1.3 s at 512 tokens, 3.4 s at 2,048, 12.3 s at 8,192, **229 s at 131,292 and 521 s at 250,223**. A 128K-token research prompt means nearly four minutes of silence before the first token; a quarter-million-token one means nearly nine. That — not throughput — is what to design around on this model. The server starts at the model's full 262,144-token window and the largest request actually sent in this session was 250,223 tokens, which is the number published as the context; it completed normally with the node at 93.5 GB. Two things are deliberately not claimed here. There is no speculative decoding: `config.json` advertises `mtp_num_hidden_layers: 1` but the released weights contain zero `mtp.*` tensors, so there is no in-checkpoint draft to enable — and the obvious external candidate has now been measured and rejected: the vendor's dense 4.5B sibling AREX-Turbo, quantized to Q4_K_M and served as a draft, decodes 15–20% *slower* than no draft at every k from 1 to 3, even at k=1 where every single drafted token was accepted. The reason is in the architecture rather than the draft: a recurrent linear-attention state cannot be partially rolled back, so the engine checkpoints and restores it around every speculative step, and that costs about as much as the extra token buys. The flag list below records the numbers. And this is a throughput recipe only — no eval was run against this quantization, so treat Q4_K_M's effect on the model's research and tool-use quality as unmeasured.

• 23.98 tok/s single-stream at a 2,048-token prompt, warm — a 122B model on one Spark
• Decode holds up: 23.99 / 23.98 / 23.66 tok/s at 512 / 2,048 / 8,192-token prompts, 17.40 at 131,292, 13.91 at 250,223
• A 250,223-token prompt served and measured — the server comes up at the model's full 262,144 window with no tuning
• 74.9 GB of GGUF against 245.1 GB of BF16, which is what makes one node enough
• ~78 GiB serving working set of 114 usable — about 36 GiB of headroom left over
• KV costs only 24 KiB/token: 36 of 48 layers are linear attention and hold no growing cache
• TTFT is the real long-context cost: 1.3 s at 512 tokens, 12.3 s at 8,192, 229 s at 131,292, 521 s at 250,223
• No speculative decoding: the config advertises an MTP head the weights do not contain
• A reasoning model — replies arrive in `reasoning_content`, and `content` is empty until it stops thinking
• Runs on one Spark; the second node is free for something else

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• llama.cpp built with `GGML_CUDA=ON` against CUDA 13 — verified on build `ee445f9` (server reports `version: 15`), which already registers the `qwen35moe` architecture
• No patches, no out-of-tree code, no custom kernels
• ~75 GB of free disk for the two GGUF shards, plus the `hf` CLI to fetch them
• Last verified 2026-07-28 on llama.cpp ee445f9 (decode re-measured on a second node; a sibling draft model was measured and rejected)

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "arex-base",
    "messages": [{"role": "user", "content": "What is 17*23? Answer briefly."}],
    "max_tokens": 512,
    "temperature": 0
  }' | python3 -m json.toolbash

curl -s -o /tmp/r.json -w '%{time_total}s total\n' \
  http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"arex-base","messages":[{"role":"user","content":"Explain gradient checkpointing."}],"max_tokens":256,"temperature":0}'

python3 -c "import json;u=json.load(open('/tmp/r.json'))['usage'];print(u)"bash

Troubleshooting

The reply is an empty string. `choices[0].message.content` is `""` and the model appears to have generated nothing.

It generated plenty — into `reasoning_content`. AREX-Base is a reasoning model and llama.cpp routes its thinking into a separate field, leaving `content` empty until the thinking block closes. If you also hit `"finish_reason": "length"`, the budget ran out mid-thought: raise `max_tokens` (512 is a reasonable floor for a short question) and read `reasoning_content` alongside `content`.

The server exits at load with an unknown-architecture error naming `qwen35moe`.

The llama.cpp binary predates support for this architecture. `grep -o QWEN35MOE ~/Dev/llama.cpp/src/llama-arch.h` in your checkout — if it prints nothing, pull and rebuild. Verified working on build `ee445f9`, which reports `version: 15`.

Decode is a few tokens per second instead of ~24.

Two usual causes, in order of likelihood. Either the build is CPU-only — check `ldd ~/Dev/llama.cpp/build/bin/llama-server | grep libcublas` returns a line, and rebuild with `-DGGML_CUDA=ON` if it does not — or the first requests after load are still faulting the 75 GB of experts in from NVMe. Send a few hundred tokens of traffic and measure again.

Passing `--speculative-config` / a draft model and seeing no speedup, or a load error about missing MTP tensors.

There is no MTP head in this checkpoint despite what `config.json` says. `mtp_num_hidden_layers: 1` is advertised, but the BF16 release's `model.safetensors.index.json` contains zero `mtp.*` entries, so the GGUF has nothing to convert. Any speculative setup here needs an external draft — BAAI's dense 4.5B AREX-Turbo shares the 248,320-token vocabulary and is the natural candidate — and its acceptance rate must be measured before you believe any throughput gain from it.

Dead end ruled out: serving this model on vLLM.

Not attempted here, and not cheap to attempt. No FP8 or NVFP4 build of AREX-Base exists on the Hub, so a vLLM recipe means quantizing 245.1 GB of BF16 yourself; the BF16 weights also do not fit two Sparks with room for KV (245.1 GB against 228 GiB usable across the pair). The GGUF route is what makes this model single-node at all. Note also that the base checkpoint is a `Qwen3_5MoeForConditionalGeneration` carrying a vision tower, which is its own vLLM complication — this recipe serves the language model only.

A very long request appears to hang, or the client times out with nothing returned.

Prefill on this model runs at roughly 450–700 tok/s and does not speed up with length, so a 131K-token prompt takes 229 s and a 250K one takes 521 s before the first token appears. That is the server working, not wedged — `tail -f ~/arex-serve.log` shows `prompt processing … progress = 0.xx` climbing. The usual cause of a failure here is a client-side timeout: raise it past 900 s for quarter-million-token prompts.

You attach a draft model with `-md` and speculative decoding never happens - decode is unchanged and nothing in the log complains.

`--spec-type` defaults to `none` on current llama.cpp builds, so `-md`, `-ngld` and `--spec-draft-n-max` are accepted and ignored. Pass `--spec-type draft-simple` (the type list is `none,draft-simple,draft-eagle3,draft-mtp,draft-dflash,ngram-simple,ngram-map-k,ngram-map-k4v,ngram-mod,ngram-cache`). The silence is total: the draft loads and is validated, the boot log carries no `spec`/`draft` line at `-lv 3`, and `common_speculative_init()` logs `no implementations specified for speculative decoding` only at TRC. Verify it is really on by reading `timings.draft_n` from a `/completion` response - llama.cpp emits `draft_n` / `draft_n_accepted` only when the speculator actually drafted, so an absent (`null`) field means it never ran, not that acceptance was zero.

Converting a Qwen3.5-family draft to GGUF succeeds, then llama.cpp refuses to load it: `error loading model: missing tensor 'blk.32.attn_norm.weight'` on a 32-layer model.

The checkpoint's `config.json` declares `mtp_num_hidden_layers: 1` while its weight index contains no `mtp.*` tensors, and `convert_hf_to_gguf.py` faithfully writes `nextn_predict_layers = 1` - so the loader then demands an extra block that was never in the file. Re-convert with `--no-mtp`. Check first with `sum('mtp' in k for k in index['weight_map'])` on `model.safetensors.index.json`; several models in this family advertise an MTP head they do not ship.