docs: how the Uncensored payload is really fetched (it is shipped in parts)
Correcting a claim I made from a bad query: Q4_K_S does exist for this finetune. `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF` ships the large quants as `.partNofM` fragments — Q4_K_M 37.00+37.00+36.97 = 110.97 GiB (which is how the deployed file was assembled) and Q4_K_S 35.00+35.00+34.09 = 104.09 GiB — while the smaller rungs are single files. A tree listing filtered on `.gguf` makes Q4_K_S invisible, which is exactly how I concluded it did not exist. Also records the fetch trap that hid it for a moment: `curl -o FILE URL` exits 0 on 401/403, so a gated repo produced three 153-byte "shards" with identical sha256 values and a cheerful DOWNLOAD_DONE in 20 s for 104 GiB. -f plus the Hub's declared size plus the LFS sha256 is the standard the fleet's own download-frank-medium-gguf.sh already meets.
This commit is contained in:
@@ -591,6 +591,23 @@ linked runbooks.
|
||||
(169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile
|
||||
upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP
|
||||
was open from that bridge — it never was).
|
||||
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as
|
||||
`.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`
|
||||
carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00
|
||||
+ 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*,
|
||||
IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS
|
||||
sha256, concatenate in numeric order — that is how the deployed 111 GiB file
|
||||
was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes
|
||||
Q4_K_S look like it does not exist.
|
||||
- **A fetch script must `-f` and verify, or it reports success on an access
|
||||
error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body
|
||||
as the "artifact": a gated repo yielded three 153-byte shards with identical
|
||||
sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use
|
||||
`-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared
|
||||
size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh`
|
||||
already does all three; ad-hoc fetches tend not to. Gated repos (e.g.
|
||||
`orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S)
|
||||
return 403 even with the host's HF token until that account accepts the gate.
|
||||
|
||||
## Runbook Index
|
||||
|
||||
|
||||
@@ -42,10 +42,19 @@ From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`,
|
||||
every 4th — `full_attention_interval = 4`) and 36 are GDN/linear.
|
||||
- Only the 12 attention layers carry KV. At the engine's measured 28 KiB per
|
||||
position, a 200k-token conversation reads **5.5 GiB of KV per token**.
|
||||
|
||||
- A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with
|
||||
every layer (75 GiB / 48). Attention layers are spread across the whole depth
|
||||
(every 4th), so no contiguous layer range can hold them all.
|
||||
|
||||
The GGUF comes from `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`, which
|
||||
ships this quant as three `.partNofM` fragments (37.00 + 37.00 + 36.97 =
|
||||
110.97 GiB) concatenated in numeric order. The same repo carries the smaller
|
||||
rungs — i1-Q4_K_S 104.09 GiB (also as parts), i1-IQ4_XS 90.78, i1-Q3_K_L 90.02,
|
||||
i1-Q3_K_M 87.67, i1-IQ3_M 83.39, i1-IQ3_S 82.78, i1-Q3_K_S 82.66, i1-Q2_K
|
||||
74.92 GiB — which is where a "does a smaller quant buy anything?" test would
|
||||
start (it buys memory, ~5 % decode, and costs quality; see the write-up below).
|
||||
|
||||
That is why the interesting split is *not* "put the experts somewhere else" —
|
||||
experts are re-read once per token but they are only ~1.7 GB of traffic — it is
|
||||
"get KV onto the 936 GB/s pool", because KV is the O(context) term.
|
||||
|
||||
Reference in New Issue
Block a user