the x4 link, measured: 6.72 GB/s — which decides the streaming question

pcie-link-test.cu (CUDA, pinned buffers, cudaEvent timing, both services stopped):
pinned H2D 6.72 GB/s and D2H 6.60 GB/s, i.e. 85 % of the Gen4 x4 ceiling. Async
is no better than sync and pageable sits within 1 % of pinned, so these copies are
link-limited rather than staging-limited — the usual "pageable costs 2x" does not
apply on this host. A 3 MiB slab costs 474 us, essentially 3 MiB / 6.72 GB/s plus
~27 us of launch.

That settles the expert-streaming question quantitatively: 1.70 GiB of active
experts per token (A3B @ Q4_K_M, 75 GiB expert mass) is 272 ms of link time ⇒
3.7 tok/s, 4.6 with a fifth of the experts resident. The same bytes computed in
host RAM at 256 GB/s cost 7.1 ms — the link is 31x slower than the memory behind
it, which is why llama.cpp's -ot exps=CPU path measured 20.46 tok/s while an
engine that streams loses by 3-5x. The port under the card is x8-capable and the
card is x16-capable, so the x4 is the adapter; doubling it still leaves streaming
well behind.

Also records FreeToken's own constraint: it keeps experts in host RAM with an LRU
GPU cache (exactly this streaming shape) and loads only modelopt block-FP8
safetensors for qwen3_5_moe, so Q4_K_M GGUF is unloadable there regardless.

AGENTS.md

@@ -415,6 +415,24 @@ linked runbooks.
   target that works, because the CPU backend is last in the list. Pure Vulkan
   changes nothing here — the gate is the override, not the driver mix. See
   [[x4-pair-experiment]].
+- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design
+  constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16,
+  services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4
+  ceiling — with async no better than sync and pageable within 1 % of pinned,
+  i.e. the copies are link-limited rather than staging-limited (the usual
+  "pageable costs 2x" rule does not apply here). The card is x16-capable and the
+  Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on
+  `00:03.1`), so the x4 lives in the adapter or cable; the link idles at
+  2.5 GT/s x4 and trains up under load. Consequence for any engine that
+  *streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B
+  model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a
+  20 % resident cache), against **20.46 tok/s** for the same model with the
+  experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link
+  carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses;
+  computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly
+  this streaming shape — also loads only modelopt block-FP8 safetensors for
+  `qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even
+  the question. See [[x4-pair-experiment]].
 
 
 **ROCm / GPU**

serverAppConfigs-docs/LLM/x4-pair-experiment.md

@@ -348,4 +348,57 @@ Nothing structural moves — `-ot` still gates per-tensor placement and a layer
 split still cannot consolidate KV on the 3090. If the aim were to *deploy*
 mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256
 `b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV
-is the lever that changes depth bandwidth; Q4_K_S is the conservative middle.
+is the lever that changes depth bandwidth; Q4_K_S is the conservative middle.
+
+## The link, measured (2026-09-16)
+
+`pcie-link-test.cu` — CUDA, pinned host buffers, `cudaEvent` timing, both
+services stopped. Gen4 x4 spec ceiling is 7.877 GB/s usable; GB/s throughout.
+
+| size | H2D sync | H2D async | D2H sync | 1× sync xfer | pageable H2D |
+|---|---|---|---|---|---|
+| 3 MiB | 6.65 | 6.71 | 6.54 | **474 µs** | 6.33 |
+| 16 MiB | 6.72 | 6.72 | 6.59 | 2.50 ms | 6.61 |
+| 64 MiB | 6.72 | 6.71 | 6.60 | 9.97 ms | 6.64 |
+| 256 MiB | 6.73 | 6.73 | 6.60 | 39.9 ms | 6.67 |
+| 1 GiB | 6.72 | 6.72 | 6.60 | 160 ms | 6.67 |
+| 2 GiB | 6.72 | 6.72 | 6.60 | 320 ms | 6.67 |
+
+Three readings:
+
+- **6.72 GB/s H2D = 85 % of the Gen4 x4 ceiling** (D2H 6.60 = 84 %). The card is
+  x16-capable and the Strix Halo port beneath it is x8-capable
+  (`LnkCap: 16GT/s, Width x8` on `00:03.1`), so the x4 lives in the
+  adapter/cable, not the slot. The link idles at 2.5 GT/s x4 and trains up under
+  load — which these numbers prove on their own, since Gen1 x4 tops out near
+  1 GB/s.
+- **Async buys nothing and pinning buys ~1 %**: one stream of copies already
+  saturates the link (6.71 async vs 6.65 sync at 3 MiB; identical at 1 GiB), and
+  pageable transfers run within noise of pinned ones. On this host the copies
+  are link-limited rather than staging-limited, so the usual "pageable costs 2×"
+  rule does not apply.
+- **A 3 MiB slab costs 474 µs**, essentially `3 MiB / 6.72 GB/s` (447 µs) plus
+  ~27 µs of launch. The streaming tax here is bandwidth, not per-copy overhead.
+
+### What that means for an expert-streaming engine
+
+Active expert bytes for this model are ~1.70 GiB/token (A3B @ Q4_K_M, 75 GiB
+expert mass) — 580 × 3 MiB slabs:
+
+| assumption | per-token link time | ceiling |
+|---|---|---|
+| every slab misses (bandwidth-bound) | 272 ms | **3.7 tok/s** |
+| 20 % resident in a 24 GiB card's expert cache | 217 ms | 4.6 tok/s |
+| 50 % hit rate (generous locality) | 136 ms | 7.4 tok/s |
+| *the same bytes computed in host RAM at 256 GB/s* | 7.1 ms | the link is **31× slower** than the memory behind it |
+
+Against this host's measured alternatives — **20.46 tok/s** for llama.cpp with
+the experts *computed* on the CPU (link carries activations only), 25 for
+halogen, 26.5 for mainline on the iGPU — an engine that *streams* experts across
+this link loses by 3–5×, and the port's x8 ceiling would only double it (≈7–9
+tok/s).
+
+For the specific engine that raised the question: FreeToken v0.1.2 keeps expert
+weights in host RAM with an LRU GPU cache — exactly this streaming shape — and
+it loads only modelopt block-FP8 HF safetensors for `qwen3_5_moe`, so a Q4_K_M
+**GGUF** is not loadable at all before bandwidth enters the discussion.