docs: how the Uncensored payload is really fetched (it is shipped in parts)

Correcting a claim I made from a bad query: Q4_K_S does exist for this finetune.
`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF` ships the large quants as
`.partNofM` fragments — Q4_K_M 37.00+37.00+36.97 = 110.97 GiB (which is how the
deployed file was assembled) and Q4_K_S 35.00+35.00+34.09 = 104.09 GiB — while
the smaller rungs are single files. A tree listing filtered on `.gguf` makes
Q4_K_S invisible, which is exactly how I concluded it did not exist.

Also records the fetch trap that hid it for a moment: `curl -o FILE URL` exits 0
on 401/403, so a gated repo produced three 153-byte "shards" with identical
sha256 values and a cheerful DOWNLOAD_DONE in 20 s for 104 GiB. -f plus the
Hub's declared size plus the LFS sha256 is the standard the fleet's own
download-frank-medium-gguf.sh already meets.
This commit is contained in:
Frank committed 2026-09-16 20:13:44 -07:00
1 parent ad18d2ddd8
commit 88f0d79021
2 files changed
+26

No files matched your search

+17
View File
@@ -591,6 +591,23 @@ linked runbooks.
(169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile (169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile
upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP
was open from that bridge — it never was). was open from that bridge — it never was).
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as
`.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`
carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00
+ 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*,
IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS
sha256, concatenate in numeric order — that is how the deployed 111 GiB file
was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes
Q4_K_S look like it does not exist.
- **A fetch script must `-f` and verify, or it reports success on an access
error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body
as the "artifact": a gated repo yielded three 153-byte shards with identical
sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use
`-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared
size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh`
already does all three; ad-hoc fetches tend not to. Gated repos (e.g.
`orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S)
return 403 even with the host's HF token until that account accepts the gate.
## Runbook Index ## Runbook Index
@@ -42,10 +42,19 @@ From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`,
every 4th — `full_attention_interval = 4`) and 36 are GDN/linear. every 4th — `full_attention_interval = 4`) and 36 are GDN/linear.
- Only the 12 attention layers carry KV. At the engine's measured 28 KiB per - Only the 12 attention layers carry KV. At the engine's measured 28 KiB per
position, a 200k-token conversation reads **5.5 GiB of KV per token**. position, a 200k-token conversation reads **5.5 GiB of KV per token**.
- A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with - A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with
every layer (75 GiB / 48). Attention layers are spread across the whole depth every layer (75 GiB / 48). Attention layers are spread across the whole depth
(every 4th), so no contiguous layer range can hold them all. (every 4th), so no contiguous layer range can hold them all.
The GGUF comes from `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`, which
ships this quant as three `.partNofM` fragments (37.00 + 37.00 + 36.97 =
110.97 GiB) concatenated in numeric order. The same repo carries the smaller
rungs — i1-Q4_K_S 104.09 GiB (also as parts), i1-IQ4_XS 90.78, i1-Q3_K_L 90.02,
i1-Q3_K_M 87.67, i1-IQ3_M 83.39, i1-IQ3_S 82.78, i1-Q3_K_S 82.66, i1-Q2_K
74.92 GiB — which is where a "does a smaller quant buy anything?" test would
start (it buys memory, ~5 % decode, and costs quality; see the write-up below).
That is why the interesting split is *not* "put the experts somewhere else" — That is why the interesting split is *not* "put the experts somewhere else" —
experts are re-read once per token but they are only ~1.7 GB of traffic — it is experts are re-read once per token but they are only ~1.7 GB of traffic — it is
"get KV onto the 936 GB/s pool", because KV is the O(context) term. "get KV onto the 936 GB/s pool", because KV is the O(context) term.