From 88f0d790215c1e3ee18225b6e744f47560d3cb8e Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 20:13:44 -0700 Subject: [PATCH] docs: how the Uncensored payload is really fetched (it is shipped in parts) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Correcting a claim I made from a bad query: Q4_K_S does exist for this finetune. `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF` ships the large quants as `.partNofM` fragments — Q4_K_M 37.00+37.00+36.97 = 110.97 GiB (which is how the deployed file was assembled) and Q4_K_S 35.00+35.00+34.09 = 104.09 GiB — while the smaller rungs are single files. A tree listing filtered on `.gguf` makes Q4_K_S invisible, which is exactly how I concluded it did not exist. Also records the fetch trap that hid it for a moment: `curl -o FILE URL` exits 0 on 401/403, so a gated repo produced three 153-byte "shards" with identical sha256 values and a cheerful DOWNLOAD_DONE in 20 s for 104 GiB. -f plus the Hub's declared size plus the LFS sha256 is the standard the fleet's own download-frank-medium-gguf.sh already meets. --- AGENTS.md | 17 +++++++++++++++++ serverAppConfigs-docs/LLM/x4-pair-experiment.md | 9 +++++++++ 2 files changed, 26 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 80a8d37..88c1d6c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -591,6 +591,23 @@ linked runbooks. (169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP was open from that bridge — it never was). +- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as + `.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF` + carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00 + + 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, + IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS + sha256, concatenate in numeric order — that is how the deployed 111 GiB file + was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes + Q4_K_S look like it does not exist. +- **A fetch script must `-f` and verify, or it reports success on an access + error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body + as the "artifact": a gated repo yielded three 153-byte shards with identical + sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use + `-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared + size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh` + already does all three; ad-hoc fetches tend not to. Gated repos (e.g. + `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S) + return 403 even with the host's HF token until that account accepts the gate. ## Runbook Index diff --git a/serverAppConfigs-docs/LLM/x4-pair-experiment.md b/serverAppConfigs-docs/LLM/x4-pair-experiment.md index 6e29d3e..3a20c48 100644 --- a/serverAppConfigs-docs/LLM/x4-pair-experiment.md +++ b/serverAppConfigs-docs/LLM/x4-pair-experiment.md @@ -42,10 +42,19 @@ From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, every 4th — `full_attention_interval = 4`) and 36 are GDN/linear. - Only the 12 attention layers carry KV. At the engine's measured 28 KiB per position, a 200k-token conversation reads **5.5 GiB of KV per token**. + - A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with every layer (75 GiB / 48). Attention layers are spread across the whole depth (every 4th), so no contiguous layer range can hold them all. +The GGUF comes from `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`, which +ships this quant as three `.partNofM` fragments (37.00 + 37.00 + 36.97 = +110.97 GiB) concatenated in numeric order. The same repo carries the smaller +rungs — i1-Q4_K_S 104.09 GiB (also as parts), i1-IQ4_XS 90.78, i1-Q3_K_L 90.02, +i1-Q3_K_M 87.67, i1-IQ3_M 83.39, i1-IQ3_S 82.78, i1-Q3_K_S 82.66, i1-Q2_K +74.92 GiB — which is where a "does a smaller quant buy anything?" test would +start (it buys memory, ~5 % decode, and costs quality; see the write-up below). + That is why the interesting split is *not* "put the experts somewhere else" — experts are re-read once per token but they are only ~1.7 GB of traffic — it is "get KV onto the 936 GB/s pool", because KV is the O(context) term.