Agents-A1 35B-A3B — FP8

InternScience's agentic-RL Agents-A1 (35B MoE, ~3B active) in the vendor's FP8 on one DGX Spark, serving the full 262,144-token context at 38.8 tok/s single-stream.

Contributors

@kvncrw

Agents-A1 is a 35B mixture-of-experts model with ~3B active parameters, hybrid attention (30 gated-delta linear-attention layers to 10 full-attention layers) and a 262,144-token window. The vendor ships an FP8 export alongside the BF16 release, and that export is the whole reason this configuration is easy: 37.7 GB in one safetensors file means a single Spark holds the weights with two thirds of its unified memory to spare, and the hybrid layer plan means the full native context costs 8 GiB of KV rather than a fight. So there is no memory puzzle to solve here, and the recipe deliberately does not invent one. The flags below are the plain ones: no tensor parallelism, no MoE backend override, no --enforce-eager, no --quantization, and specifically no --language-model-only. What the configuration is actually shaped by is the checkpoint's quantization scope. The vendor quantized the routed experts and the full-attention projections to FP8 and left the linear-attention path, the routers, the embedding table and lm_head in BF16 — which is a reasonable cut, but it means the per-token read is not dominated by the thing the file is dominated by. Summing the header, a decode step reads about 4.49 GB: 1.01 GB of routed experts (8 of 256), 2.02 GB of linear-attention projections and 1.02 GB of unquantized lm_head. The experts are 86% of the file and 22% of the reads. That ratio is what sets the measured 38.8 tok/s, and it is also why a 4-bit build of the same architecture is much faster than this one rather than slightly faster. The other thing this recipe settles is the speculative-decoding question, because the obvious answer is wrong. The vendor publishes a dense 4.5B sibling on the identical 248,320-token vocabulary, which looks like a ready-made draft. It is not: quantized to NVFP4 that sibling still reads 3.52 GB per token against the target's 4.49 GB, so a drafted token costs 78% of a verified one. Even at 100% acceptance and k=1 the arithmetic ceiling is about +12%, and at any realistic acceptance rate it is a loss. Sparsity inverts the usual intuition — an 8x smaller model is not an 8x cheaper draft when the big one only reads 3% of its experts. This configuration therefore ships with no draft, and the section below records both the arithmetic and the load failure we hit trying it anyway.

• 1 × DGX Spark (GB10, sm_121) — no tensor parallelism, no second node
• Measured 38.77 tok/s single-stream decode at a 2,048-token prompt, warm, greedy, on stock vLLM 0.26.0
• Full 262,144-token context served, bracketed with real requests: a 253,840-token prompt answers in 105.3 s, and anything past the window is refused with HTTP 400 rather than crashing
• Weights 35.31 GiB; whole serving working set ~54 of 114 usable GiB
• 8 GiB of KV buys 412,980 tokens — 1.58x concurrency at the full window, because only 10 of 40 layers cache
• No flags beyond the defaults: vLLM auto-selects the TRITON FP8 MoE backend and FLASH_ATTN on sm_121, and both work
• A decode step reads 4.49 GB, of which the 256 routed experts are only 1.01 GB — measured from the safetensors header
• No speculative draft: the config advertises mtp_num_hidden_layers: 1 but the checkpoint ships zero MTP tensors, and the vendor's dense 4.5B sibling is too expensive per token to pay as an external draft
• Weight load 282 s from one 37.7 GB safetensors file; full startup about 7 minutes

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.26.0 with torch 2.11.0+cu130 (verified 2026-07-28). Stock, unpatched
• The `hf` CLI to fetch the checkpoint (~37.7 GB)
• ~45 GB of free disk on the serving node

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Agents-A1-FP8",
       "messages":[{"role":"user","content":"Refactor this function to be tail-recursive."}],
       "max_tokens":512,"temperature":0}'bash

curl -s -N http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Agents-A1-FP8",
       "messages":[{"role":"user","content":"Write a red-black tree in Python."}],
       "max_tokens":256,"temperature":0,
       "stream":true,"stream_options":{"include_usage":true}}' \
  | tail -2bash

Troubleshooting

The dense 4.5B sibling refuses to load as an external draft: AssertionError in load_column_parallel_weight

Do not run it. The failure is fixable in principle (serve a draft whose files match one format), but it is not worth fixing here — see the next entry for why the pairing cannot pay even if it loads.

You want speculative decoding on this model and there is an obvious candidate draft

Screen a draft by bytes-read-per-token, not by parameter count, before spending a download on it. At k=1 the best case is (1 + 0.78) reads for 2 tokens, an 11% ceiling at 100% acceptance; realistic acceptance makes it a loss, and larger k makes it worse. A useful draft on this hardware wants to be well under ~15% of the target's per-token read — which for a sparse MoE target usually means an MTP head or a purpose-built EAGLE/DFlash draft, not a small dense sibling.

A prompt longer than the window is rejected with HTTP 400

Nothing to fix — but note the shape of the curve before promising this context to anyone. Time-to-first-token runs 196 ms at 511 tokens, 403 ms at 2,042, 1.35 s at 8,208, 43.1 s at 144,950 and 105.3 s at 253,840, while decode stays between 37.7 and 39.0 tok/s across the short end. The long-context story here is a prefill cost, not a decode cost.

Decode is 38.8 tok/s and you expected more from a 3B-active MoE

Nothing is misconfigured. If you want this architecture faster on a Spark, the lever is a quant that also compresses the linear-attention projections and lm_head, not a different MoE backend or a bigger KV pool. Diff a candidate quant's ignore list before assuming lower bits mean faster decode.

vLLM picks the TRITON FP8 MoE backend and you want to force something else

Leave it on auto. MARLIN was measured against the auto-selected TRITON on the same node in the same session, warm, and the two are not separable: 39.08 vs 38.77 tok/s decode and 4,945 vs 5,071 tok/s prefill at a 2,042-token prompt, both differences well inside the boot-to-boot spread on this hardware. Memory is identical too (35.25 vs 35.31 GiB loaded, the same 412,980-token pool). There is no backend win to chase here — the ceiling is the checkpoint's quantization scope, not the kernel.