Can you move the speculative drafter onto another machine?

<!-- Draft. Numbers live in data/case-studies.json (runs[]) and drive the charts — keep them out of this prose so the two can't disagree. Delete this note at publish. -->

Background

We have two DGX Sparks running GLM-5.2 (753B MoE, TP2) with a dspark speculative drafter co-located on them, and a separate box with two idle RTX 5090s. The tempting move: relocate the drafter onto the 5090s. The Sparks are memory-bandwidth-starved; the 5090s have bandwidth to spare and 64 GB of GDDR7 doing nothing. Free the Sparks of draft work, maybe even run a bigger drafter that pushes acceptance up.

The catch is that a speculative cycle becomes a network round trip. So the question isn't "is the 5090 faster at drafting" — obviously it is — it's whether the wire is cheap enough that the answer matters.

Method

What has to cross the wire

This is where the intuition "it's only a few token ids, that's nothing" breaks down. A dspark drafter isn't conditioned on tokens — it's conditioned on the target model's aux hidden states, pulled from layers [8, 23, 39, 55, 70].

Per token, that's 5 × 6144 × 2 bytes (BF16). Multiply by the tokens accepted per cycle and you get the real per-cycle payload. The return trip (K draft token ids) genuinely is nothing.

<!-- TODO: state the measured payload and cycle time here once the runs land — the chart carries the numbers. -->

The rungs

Same payload, same cycle, three link speeds spanning ~2000×:

All three measured with hopbench (stdlib TCP, TCP_NODELAY, real request/response shape rather than bulk streaming — an iperf3 number would answer the wrong question).

Findings

Bandwidth-bound or latency-bound?

If hop cost falls off a cliff between rungs, the transfer is bandwidth-bound and a faster link fixes it. If it's flat, it's latency-bound and no amount of bandwidth helps. The answer turned out to be both, depending where you stand: the two Ethernet rungs are cleanly bandwidth-bound (latency linear in payload, wire saturated), and only the RoCE fabric goes latency-bound. So the crossover sits between 1 GbE and the fabric — not, as I'd assumed, between 100 Mb and 1 GbE.

Why the answer probably isn't "just move it"

Two results already on this stack constrain what relocation could possibly buy:

1. The draft's scheduling was never the cost — but its bandwidth might be. MTP-DRAFT-COST.md (session 12) records Approach A — wrapping the drafter in a FULL cudagraph — as built, confirmed engaged, and refuted: 18.5 → 18.5 tok/s. It's tempting to read that as "the draft is free, so relocating it frees nothing." That reading is wrong: the cudagraph removed host launch gaps, not the drafter's weight reads. On a machine whose binding constraint is memory bandwidth, those reads are the cost.

   Measured directly, t_draft = 15.88 ms — draft_fwd 9.91 + markov_sample 4.75 + ctxkv 1.23, from the fork's own VLLM_DSPARK_PROFILE hooks. That's 17.8% of the 89.34 ms cycle at the K=2 optimum, and draft_fwd alone lands within 23% of the 12.8 ms you'd predict from just reading the speculator's 3.5 GiB/rank at ~273 GB/s. The draft really is a weight read, and moving it off-box frees that bandwidth.

   Getting here required care. Inferring t_draft from the spec-depth sweep's intercept gave 2.8 ± 9.3 ms — worthless, because run-to-run scatter exceeded the quantity (see the depth study). And the first direct profile read ctxkv at 362 ms inside an 89 ms cycle, because the leading cuda.synchronize() of each step was absorbing the previous step's verify forward. Draining the queue before the timer starts fixed it.

2. The acceptance ceiling isn't a capacity problem. Four independently trained same-architecture drafts — including a from-scratch 4B — all converge at pos-0 ≈ 0.72–0.75. Sparkulator v4, which is exactly the "richer conditioning" idea, was built and failed GATE B. Confidence-gated dynamic K measured NULL. A bigger drafter on a bigger GPU doesn't obviously address an information bottleneck.

Which means the interesting version of this question isn't the synchronous one.

The verdict

Put the two measured numbers side by side and the question answers itself:

freed by relocating the drafter :  15.88 ms
cost of the hop                 :   0.198 ms  (200 Gb RoCE)
                                    1.713 ms  (1 GbE)
                                   16.02  ms  (100 Mb Ethernet)

The break-even hop is 15.88 ms. Over gigabit the hop costs about a ninth of what it frees, projecting 24.05 → 28.59 tok/s (+19%); over the RoCE fabric, +21%. And 100 Mb Ethernet costs 16.02 ms — a dead heat with break-even, to within 1%. The single link this GPU box actually had was, by coincidence, precisely the point where the idea stops being worth doing.

Limitations

Two caveats worth keeping attached to those projections. They assume the whole 15.88 ms is genuinely freed and nothing else stalls — the aux hidden states are already computed as part of the verify forward, so that part is safe, but no remote proposer exists to serve, so this remains an extrapolation from two measurements rather than an end-to-end result. And t_draft was measured under profiling, whose extra syncs suppress any overlap that might exist in the unprofiled engine, making 15.88 ms an upper bound on what is truly serialized.

<!-- TODO: fold in the two-NIC discovery — the GPU rig's 5 GbE Realtek port has no driver bound on kernel 6.8, so a cable plugged into it is electrically dark — plus the admin-down Spark port, and the link-partner misattribution that briefly convinced me there was a 10/100 switch in the path. See docs/dgx-spark-notes.md. (The boring finding that cost an afternoon.) -->

Open questions

The remaining obstacle is engineering, not physics: dspark's proposer runs in-process in the Ray TP workers, and the only hidden-state export path writes safetensors to disk for offline distillation. A synchronous remote proposer is a real vLLM port. But it is now a port with a measured payoff attached to it.