diff --git a/AGENTS.md b/AGENTS.md index 43e0953..357c005 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -415,6 +415,24 @@ linked runbooks. target that works, because the CPU backend is last in the list. Pure Vulkan changes nothing here — the gate is the override, not the driver mix. See [[x4-pair-experiment]]. +- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design + constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16, + services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4 + ceiling — with async no better than sync and pageable within 1 % of pinned, + i.e. the copies are link-limited rather than staging-limited (the usual + "pageable costs 2x" rule does not apply here). The card is x16-capable and the + Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on + `00:03.1`), so the x4 lives in the adapter or cable; the link idles at + 2.5 GT/s x4 and trains up under load. Consequence for any engine that + *streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B + model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a + 20 % resident cache), against **20.46 tok/s** for the same model with the + experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link + carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses; + computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly + this streaming shape — also loads only modelopt block-FP8 safetensors for + `qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even + the question. See [[x4-pair-experiment]]. **ROCm / GPU** diff --git a/serverAppConfigs-docs/LLM/x4-pair-experiment.md b/serverAppConfigs-docs/LLM/x4-pair-experiment.md index 798136f..16d05fb 100644 --- a/serverAppConfigs-docs/LLM/x4-pair-experiment.md +++ b/serverAppConfigs-docs/LLM/x4-pair-experiment.md @@ -348,4 +348,57 @@ Nothing structural moves — `-ot` still gates per-tensor placement and a layer split still cannot consolidate KV on the 3090. If the aim were to *deploy* mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256 `b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV -is the lever that changes depth bandwidth; Q4_K_S is the conservative middle. \ No newline at end of file +is the lever that changes depth bandwidth; Q4_K_S is the conservative middle. + +## The link, measured (2026-09-16) + +`pcie-link-test.cu` — CUDA, pinned host buffers, `cudaEvent` timing, both +services stopped. Gen4 x4 spec ceiling is 7.877 GB/s usable; GB/s throughout. + +| size | H2D sync | H2D async | D2H sync | 1× sync xfer | pageable H2D | +|---|---|---|---|---|---| +| 3 MiB | 6.65 | 6.71 | 6.54 | **474 µs** | 6.33 | +| 16 MiB | 6.72 | 6.72 | 6.59 | 2.50 ms | 6.61 | +| 64 MiB | 6.72 | 6.71 | 6.60 | 9.97 ms | 6.64 | +| 256 MiB | 6.73 | 6.73 | 6.60 | 39.9 ms | 6.67 | +| 1 GiB | 6.72 | 6.72 | 6.60 | 160 ms | 6.67 | +| 2 GiB | 6.72 | 6.72 | 6.60 | 320 ms | 6.67 | + +Three readings: + +- **6.72 GB/s H2D = 85 % of the Gen4 x4 ceiling** (D2H 6.60 = 84 %). The card is + x16-capable and the Strix Halo port beneath it is x8-capable + (`LnkCap: 16GT/s, Width x8` on `00:03.1`), so the x4 lives in the + adapter/cable, not the slot. The link idles at 2.5 GT/s x4 and trains up under + load — which these numbers prove on their own, since Gen1 x4 tops out near + 1 GB/s. +- **Async buys nothing and pinning buys ~1 %**: one stream of copies already + saturates the link (6.71 async vs 6.65 sync at 3 MiB; identical at 1 GiB), and + pageable transfers run within noise of pinned ones. On this host the copies + are link-limited rather than staging-limited, so the usual "pageable costs 2×" + rule does not apply. +- **A 3 MiB slab costs 474 µs**, essentially `3 MiB / 6.72 GB/s` (447 µs) plus + ~27 µs of launch. The streaming tax here is bandwidth, not per-copy overhead. + +### What that means for an expert-streaming engine + +Active expert bytes for this model are ~1.70 GiB/token (A3B @ Q4_K_M, 75 GiB +expert mass) — 580 × 3 MiB slabs: + +| assumption | per-token link time | ceiling | +|---|---|---| +| every slab misses (bandwidth-bound) | 272 ms | **3.7 tok/s** | +| 20 % resident in a 24 GiB card's expert cache | 217 ms | 4.6 tok/s | +| 50 % hit rate (generous locality) | 136 ms | 7.4 tok/s | +| *the same bytes computed in host RAM at 256 GB/s* | 7.1 ms | the link is **31× slower** than the memory behind it | + +Against this host's measured alternatives — **20.46 tok/s** for llama.cpp with +the experts *computed* on the CPU (link carries activations only), 25 for +halogen, 26.5 for mainline on the iGPU — an engine that *streams* experts across +this link loses by 3–5×, and the port's x8 ceiling would only double it (≈7–9 +tok/s). + +For the specific engine that raised the question: FreeToken v0.1.2 keeps expert +weights in host RAM with an LRU GPU cache — exactly this streaming shape — and +it loads only modelopt block-FP8 HF safetensors for `qwen3_5_moe`, so a Q4_K_M +**GGUF** is not loadable at all before bandwidth enters the discussion. \ No newline at end of file