From 2f012fba39036d80ccabb5502b33b541e28a3016 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 23:25:06 -0700 Subject: [PATCH] =?UTF-8?q?the=20x4=20link,=20measured:=206.72=20GB/s=20?= =?UTF-8?q?=E2=80=94=20which=20decides=20the=20streaming=20question?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pcie-link-test.cu (CUDA, pinned buffers, cudaEvent timing, both services stopped): pinned H2D 6.72 GB/s and D2H 6.60 GB/s, i.e. 85 % of the Gen4 x4 ceiling. Async is no better than sync and pageable sits within 1 % of pinned, so these copies are link-limited rather than staging-limited — the usual "pageable costs 2x" does not apply on this host. A 3 MiB slab costs 474 us, essentially 3 MiB / 6.72 GB/s plus ~27 us of launch. That settles the expert-streaming question quantitatively: 1.70 GiB of active experts per token (A3B @ Q4_K_M, 75 GiB expert mass) is 272 ms of link time ⇒ 3.7 tok/s, 4.6 with a fifth of the experts resident. The same bytes computed in host RAM at 256 GB/s cost 7.1 ms — the link is 31x slower than the memory behind it, which is why llama.cpp's -ot exps=CPU path measured 20.46 tok/s while an engine that streams loses by 3-5x. The port under the card is x8-capable and the card is x16-capable, so the x4 is the adapter; doubling it still leaves streaming well behind. Also records FreeToken's own constraint: it keeps experts in host RAM with an LRU GPU cache (exactly this streaming shape) and loads only modelopt block-FP8 safetensors for qwen3_5_moe, so Q4_K_M GGUF is unloadable there regardless. --- AGENTS.md | 18 ++++++ .../LLM/x4-pair-experiment.md | 55 ++++++++++++++++++- 2 files changed, 72 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 43e0953..357c005 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -415,6 +415,24 @@ linked runbooks. target that works, because the CPU backend is last in the list. Pure Vulkan changes nothing here — the gate is the override, not the driver mix. See [[x4-pair-experiment]]. +- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design + constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16, + services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4 + ceiling — with async no better than sync and pageable within 1 % of pinned, + i.e. the copies are link-limited rather than staging-limited (the usual + "pageable costs 2x" rule does not apply here). The card is x16-capable and the + Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on + `00:03.1`), so the x4 lives in the adapter or cable; the link idles at + 2.5 GT/s x4 and trains up under load. Consequence for any engine that + *streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B + model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a + 20 % resident cache), against **20.46 tok/s** for the same model with the + experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link + carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses; + computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly + this streaming shape — also loads only modelopt block-FP8 safetensors for + `qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even + the question. See [[x4-pair-experiment]]. **ROCm / GPU** diff --git a/serverAppConfigs-docs/LLM/x4-pair-experiment.md b/serverAppConfigs-docs/LLM/x4-pair-experiment.md index 798136f..16d05fb 100644 --- a/serverAppConfigs-docs/LLM/x4-pair-experiment.md +++ b/serverAppConfigs-docs/LLM/x4-pair-experiment.md @@ -348,4 +348,57 @@ Nothing structural moves — `-ot` still gates per-tensor placement and a layer split still cannot consolidate KV on the 3090. If the aim were to *deploy* mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256 `b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV -is the lever that changes depth bandwidth; Q4_K_S is the conservative middle. \ No newline at end of file +is the lever that changes depth bandwidth; Q4_K_S is the conservative middle. + +## The link, measured (2026-09-16) + +`pcie-link-test.cu` — CUDA, pinned host buffers, `cudaEvent` timing, both +services stopped. Gen4 x4 spec ceiling is 7.877 GB/s usable; GB/s throughout. + +| size | H2D sync | H2D async | D2H sync | 1× sync xfer | pageable H2D | +|---|---|---|---|---|---| +| 3 MiB | 6.65 | 6.71 | 6.54 | **474 µs** | 6.33 | +| 16 MiB | 6.72 | 6.72 | 6.59 | 2.50 ms | 6.61 | +| 64 MiB | 6.72 | 6.71 | 6.60 | 9.97 ms | 6.64 | +| 256 MiB | 6.73 | 6.73 | 6.60 | 39.9 ms | 6.67 | +| 1 GiB | 6.72 | 6.72 | 6.60 | 160 ms | 6.67 | +| 2 GiB | 6.72 | 6.72 | 6.60 | 320 ms | 6.67 | + +Three readings: + +- **6.72 GB/s H2D = 85 % of the Gen4 x4 ceiling** (D2H 6.60 = 84 %). The card is + x16-capable and the Strix Halo port beneath it is x8-capable + (`LnkCap: 16GT/s, Width x8` on `00:03.1`), so the x4 lives in the + adapter/cable, not the slot. The link idles at 2.5 GT/s x4 and trains up under + load — which these numbers prove on their own, since Gen1 x4 tops out near + 1 GB/s. +- **Async buys nothing and pinning buys ~1 %**: one stream of copies already + saturates the link (6.71 async vs 6.65 sync at 3 MiB; identical at 1 GiB), and + pageable transfers run within noise of pinned ones. On this host the copies + are link-limited rather than staging-limited, so the usual "pageable costs 2×" + rule does not apply. +- **A 3 MiB slab costs 474 µs**, essentially `3 MiB / 6.72 GB/s` (447 µs) plus + ~27 µs of launch. The streaming tax here is bandwidth, not per-copy overhead. + +### What that means for an expert-streaming engine + +Active expert bytes for this model are ~1.70 GiB/token (A3B @ Q4_K_M, 75 GiB +expert mass) — 580 × 3 MiB slabs: + +| assumption | per-token link time | ceiling | +|---|---|---| +| every slab misses (bandwidth-bound) | 272 ms | **3.7 tok/s** | +| 20 % resident in a 24 GiB card's expert cache | 217 ms | 4.6 tok/s | +| 50 % hit rate (generous locality) | 136 ms | 7.4 tok/s | +| *the same bytes computed in host RAM at 256 GB/s* | 7.1 ms | the link is **31× slower** than the memory behind it | + +Against this host's measured alternatives — **20.46 tok/s** for llama.cpp with +the experts *computed* on the CPU (link carries activations only), 25 for +halogen, 26.5 for mainline on the iGPU — an engine that *streams* experts across +this link loses by 3–5×, and the port's x8 ceiling would only double it (≈7–9 +tok/s). + +For the specific engine that raised the question: FreeToken v0.1.2 keeps expert +weights in host RAM with an LRU GPU cache — exactly this streaming shape — and +it loads only modelopt block-FP8 HF safetensors for `qwen3_5_moe`, so a Q4_K_M +**GGUF** is not loadable at all before bandwidth enters the discussion. \ No newline at end of file