From f99b698d252d7ccf7e8c9cd18769462695452f44 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 8 Oct 2026 23:44:13 -0700 Subject: [PATCH] =?UTF-8?q?docs(LLM):=203090=20worked=20example=20?= =?UTF-8?q?=E2=80=94=20archon=20w4dn8=20methodology=20ports=20to=20GGUF-na?= =?UTF-8?q?tive=20quants;=2024=20GiB=20size=20law=20for=2035B-A3B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../LLM/Quantization Production.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/serverAppConfigs-docs/LLM/Quantization Production.md b/serverAppConfigs-docs/LLM/Quantization Production.md index 3a988a9..d119b38 100644 --- a/serverAppConfigs-docs/LLM/Quantization Production.md +++ b/serverAppConfigs-docs/LLM/Quantization Production.md @@ -151,6 +151,34 @@ Caveats before any swap: the serving contract (OpenAI server, 200,704 ×2 slots) must be met by the challenger runtime, and the 24 GB card runs ONE server — benching follows the stop-and-mask law. +#### Worked example — the frank-archon w4dn8 recipe ported to the 3090 +(audit 2026-10-08, target `Qwen3.6-35B-A3B`) +**The methodology ports; the encoding must be translated.** llama.cpp +decodes no GPTQ — the recipe re-expresses on GGUF-native terms: +imatrix on our corpora (replaces the radiance grams; runs the 65 GiB +BF16 on host CPU), IQ4-class experts, hot-expert protection as +`--tensor-type "=f16"` patterns over the derive-the-hot-experts law +(statistic: imatrix importances), Q6_K trunk, KLD/perplexity gate at +the deployed slot shape. Production stays host-side; the 3090 is only +the decode target. +KV is cheap on this arch: 10 full layers × 2 KV heads × 256 = +**20 KiB/token f16** (3.8 GiB per 200k slot); DeltaNet state is +per-slot, tens of MiB. Size verdict (weights ≈ 32.2 B experts + 3 B +trunk): +- Archon-faithful (w4 experts + bf16 protected + i8 trunk) ≈ 22 GiB → + 27 GiB with KV+buffers → **over 24; that shape is 2×3090-class.** +- Fits 1×200,704 text-only: IQ4_XS experts (17.4) + Q6_K trunk + q8_0 + KV (1.9; `GGML_CUDA_FA_ALL_QUANTS=ON` build) ≈ 21.5 ✓ +- Compaction-hub shape (2×200,704 + mmproj): experts at IQ3_M (14.4) ≈ + 24.0 — the exact Swift IQ3_XXS 2-slot edge, MTP precedent exists. +- Literal GPTQ-w4 encoding decodes only via vLLM/SGLang Marlin (Playbook + B); compressed-tensors expresses the protected set per-module. +Shelf check: HauhauCS publishes this model's K_P ladder (IQ2_M 10.86, +IQ4_XS 17.44, Q4_K_M 19.71, Q4_K_P 21.82, Q5_K_P 26.1, Q6_K_P 28.5, +Q8_K_P 40.6 GiB + mmproj 0.84) — Q4_K_P IS the protected-tensor blend. +The self-made delta: calibration on OUR corpora, our hot-expert law, +and a KLD gate scored at the serving shape. + ### R9700 — frank-primary (radiance) The full self-quant pipeline already exists end to end: fetch BF16 → record KLD reference (1.0.13) → capture calibration grams (1.2.0 tap,