diff --git a/AGENTS.md b/AGENTS.md index 80a8d37..88c1d6c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -591,6 +591,23 @@ linked runbooks. (169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP was open from that bridge — it never was). +- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as + `.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF` + carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00 + + 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, + IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS + sha256, concatenate in numeric order — that is how the deployed 111 GiB file + was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes + Q4_K_S look like it does not exist. +- **A fetch script must `-f` and verify, or it reports success on an access + error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body + as the "artifact": a gated repo yielded three 153-byte shards with identical + sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use + `-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared + size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh` + already does all three; ad-hoc fetches tend not to. Gated repos (e.g. + `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S) + return 403 even with the host's HF token until that account accepts the gate. ## Runbook Index diff --git a/serverAppConfigs-docs/LLM/x4-pair-experiment.md b/serverAppConfigs-docs/LLM/x4-pair-experiment.md index 6e29d3e..3a20c48 100644 --- a/serverAppConfigs-docs/LLM/x4-pair-experiment.md +++ b/serverAppConfigs-docs/LLM/x4-pair-experiment.md @@ -42,10 +42,19 @@ From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, every 4th — `full_attention_interval = 4`) and 36 are GDN/linear. - Only the 12 attention layers carry KV. At the engine's measured 28 KiB per position, a 200k-token conversation reads **5.5 GiB of KV per token**. + - A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with every layer (75 GiB / 48). Attention layers are spread across the whole depth (every 4th), so no contiguous layer range can hold them all. +The GGUF comes from `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`, which +ships this quant as three `.partNofM` fragments (37.00 + 37.00 + 36.97 = +110.97 GiB) concatenated in numeric order. The same repo carries the smaller +rungs — i1-Q4_K_S 104.09 GiB (also as parts), i1-IQ4_XS 90.78, i1-Q3_K_L 90.02, +i1-Q3_K_M 87.67, i1-IQ3_M 83.39, i1-IQ3_S 82.78, i1-Q3_K_S 82.66, i1-Q2_K +74.92 GiB — which is where a "does a smaller quant buy anything?" test would +start (it buys memory, ~5 % decode, and costs quality; see the write-up below). + That is why the interesting split is *not* "put the experts somewhere else" — experts are re-read once per token but they are only ~1.7 GB of traffic — it is "get KV onto the 936 GB/s pool", because KV is the O(context) term.