LFM2.5 8B-A1B (Liquid AI) — BF16

Liquid AI's LFM2.5 8B-A1B (8.4B MoE, ~1B active) unquantized in BF16 on one DGX Spark, serving the full 128,000-token context at 57.5 tok/s single-stream.

Contributors

@kvncrw@Marinara_Waif

LFM2.5 8B-A1B is deliberately the small end of what a DGX Spark can run, and it is worth a page for two reasons that have nothing to do with size. The first is the sequence mixer: 18 of its 24 layers are short convolutions (conv_L_cache 3) and only 6 are full attention, so the KV cache costs 12.29 KiB per token instead of the 40-80 KiB a same-size dense transformer would want. The model's entire 128,000-token native window fits in a 3 GiB pool with headroom to spare — there is no context tuning to do here, only a decision about how much of the pool to hand back. The second is what that buys you in practice: a 126,026-token prompt answers in 13.5 seconds on one node, and decode only falls from 57.5 tok/s at a 2,048-token prompt to 44.9 at 98,528 — a 22% drop across a 48x range of prompt length. The honest counterweight is that 57.5 tok/s is not the runaway number the parameter count suggests. Decode on a Spark is bandwidth-bound, and summing this checkpoint's per-token reads gives 3.37 GB: 1,938 MB of routed experts (top-4 of 32), 604 MB of short-conv projections, 524 MB of tied lm_head, and 302 MB of everything else. Against the Spark's 273 GB/s that is an 81 tok/s ceiling, and the measured 57.5 sits at 71% of it — a healthy, well-configured fraction, not a misconfiguration. An 8B MoE is small in memory, not proportionally small in bytes read per token. Reasoning is always on: the chat template exposes no thinking toggle, so every response opens with a <think> block that counts against your output budget.

• One DGX Spark, no tensor parallelism, no quantization step — the vendor BF16 checkpoint served as-is
• 57.5 tok/s single-stream decode at a 2,048-token prompt (vLLM 0.26.0, warm, greedy)
• The full 128,000-token native context, verified with a 126,026-token request that answered in 13.5 s
• 24.7 GiB of serving working set on a 114 GiB node — 16.9 GB of weights plus a 3 GiB KV pin
• Only 6 of 24 layers are full attention, so KV costs 12.29 KiB/token and 128K context is effectively free
• Prefill peaks at 14,600 tok/s; TTFT is 192 ms at 2K, 2.4 s at 32K, 9.6 s at 98K
• No speculative decoding — the checkpoint ships no MTP head and no EAGLE/DFlash draft exists for it

Software requirements

• NVIDIA driver 580.159.03 (CUDA 13 capable) — the stock DGX Spark image
• vLLM 0.26.0 with torch 2.11.0+cu130. Lfm2MoeForCausalLM is registered natively; no patch, no --trust-remote-code, no out-of-tree plugin
• vLLM 0.24.0 also registers the architecture if that is what you have installed; the numbers on this page were measured on 0.26.0
• Last verified 2026-07-27 on vLLM 0.26.0
• ~17 GB of free disk for the checkpoint

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "LiquidAI/LFM2.5-8B-A1B",
    "messages": [{"role": "user", "content": "Write a Python function that merges two sorted lists."}],
    "max_tokens": 512,
    "temperature": 0
  }'bash

# Take tok/s from the server's own usage accounting, not by counting SSE frames.
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"LiquidAI/LFM2.5-8B-A1B","messages":[{"role":"user","content":"Explain how a B-tree insert works, step by step."}],"max_tokens":512,"temperature":0}' \
  -w '\ntotal=%{time_total}s\n' \
  | python3 -c 'import sys,json; d=json.loads(sys.stdin.read().split(chr(10))[0]); print(d["usage"])'bash

Troubleshooting

Decode is ~57 tok/s and you expected far more from an 8B model.

That is the correct number for this checkpoint on this hardware, not a misconfiguration. Decode on a Spark is bandwidth-bound, so what matters is bytes read per token, not parameter count. Summing this checkpoint's safetensors headers gives 3.37 GB read per token: 1,938 MB of routed experts (top-4 of 32 across 22 layers), 604 MB of short-conv projections, 524 MB of tied lm_head, 176 MB of dense FFN and 126 MB of attention. Against 273 GB/s of LPDDR that is an 81 tok/s ceiling and the measured 57.5 is 71% of it. Two of those terms surprise people: the short-conv layers are cheap in KV but not in bytes, and tie_word_embeddings makes the 524 MB embedding matrix a per-token read as lm_head.

A request with a ~133,000-token prompt returns HTTP 400.

Expected — that is above --max-model-len 128000, which is the model's native maximum, and vLLM refuses it cleanly rather than crashing. The largest prompt verified on this config is 126,026 tokens, which answered in 13.5 s. If you need more you would need RoPE scaling, which this recipe does not use and has not measured.

The engine dies right after `torch.compile took NN s` with no traceback and no OOM in the kernel log.

On vLLM 0.26.0 this is usually the memory-profiling assertion, not a crashed kernel: profiling requires free memory only to decrease, which is false on unified memory where the reading drifts upward when the OS reclaims page cache. Passing --kv-cache-memory-bytes (as this recipe does) early-returns before that assert and logs 'skipped memory profiling'. Also do not poll the box with `free`, `du` or `ps` while it is profiling — that churn can cause it.

Startup seems to hang for several minutes between weight loading and the KV cache line.

Normal on a first boot. Our cold boot took 94 s to load weights, then ~6 minutes before `GPU KV cache size` appeared, then 55 s of graph capture — flashinfer has no prebuilt sm_121a cubins and compiles them on first use. Confirm it is working rather than wedged with `pgrep -af 'ninja|cicc|nvcc'`. It is cached under ~/.cache/flashinfer afterwards; keep VLLM_CACHE_ROOT stable across boots so you only pay it once.

You want to quantize it to get more speed.

Ruled out, and worth explaining. Only MLX and GGUF community quants exist — there is no NVFP4 or FP8 build of this checkpoint. More importantly the payoff is capped by shape: 15.5 of the 16.9 GB is routed experts, but top-4-of-32 routing means one token reads only 4/32 of them, so a 4-bit expert-only cut would free ~11 GiB of memory on a node that is already using 24.7 of 114, while cutting only ~970 MB from a 3.37 GB per-token read. The memory win and the speed win are different numbers here and neither is the constraint.

Speculative decoding does not start.

There is nothing to point it at. The checkpoint contains no MTP or next-token-prediction layers and no EAGLE/DFlash draft has been published for LFM2.5. Weightless prompt-lookup (--speculative-config '{"method":"ngram",...}') is the only option that needs no draft, and it is sharply workload-dependent: it only pays on traffic where the output repeats the prompt, such as apply-this-diff editing. It was not measured on this config.