pcie-link-test.cu (CUDA, pinned buffers, cudaEvent timing, both services stopped):
pinned H2D 6.72 GB/s and D2H 6.60 GB/s, i.e. 85 % of the Gen4 x4 ceiling. Async
is no better than sync and pageable sits within 1 % of pinned, so these copies are
link-limited rather than staging-limited — the usual "pageable costs 2x" does not
apply on this host. A 3 MiB slab costs 474 us, essentially 3 MiB / 6.72 GB/s plus
~27 us of launch.
That settles the expert-streaming question quantitatively: 1.70 GiB of active
experts per token (A3B @ Q4_K_M, 75 GiB expert mass) is 272 ms of link time ⇒
3.7 tok/s, 4.6 with a fifth of the experts resident. The same bytes computed in
host RAM at 256 GB/s cost 7.1 ms — the link is 31x slower than the memory behind
it, which is why llama.cpp's -ot exps=CPU path measured 20.46 tok/s while an
engine that streams loses by 3-5x. The port under the card is x8-capable and the
card is x16-capable, so the x4 is the adapter; doubling it still leaves streaming
well behind.
Also records FreeToken's own constraint: it keeps experts in host RAM with an LRU
GPU cache (exactly this streaming shape) and loads only modelopt block-FP8
safetensors for qwen3_5_moe, so Q4_K_M GGUF is unloadable there regardless.
Frank
2f012fba39