diff --git a/serverAppConfigs-docs/LLM/Quantization Production.md b/serverAppConfigs-docs/LLM/Quantization Production.md index 9d301d4..3a988a9 100644 --- a/serverAppConfigs-docs/LLM/Quantization Production.md +++ b/serverAppConfigs-docs/LLM/Quantization Production.md @@ -162,6 +162,35 @@ REAP the expert set BEFORE step 3; ablate BEFORE calibration. Native encodings beyond w4: i8 trunk, fp8 HC/PLE/KV, bf16 protected, u2 draft — all recipe-expressible (see the w4dn8 runbook). +#### Recipe portability: the w4dn8 shape onto other MoEs (2026-10-08 audit) +The pin 1.0.13 container ships SIX arch plugins +(`/opt/radiance/share/radiance/architectures/`): `qwen4exp_fp8`, +**`qwen35moe_fp8`**, `qwen35_fp8`, `qwen35_bf16`, `llama_fp8`, +`llama_dense_fp8` (census via `podman run --rm --entrypoint sh -c` +— the image has no `ls`/`strings`; grep the `.so` files directly, and +the image lives in ROOTLESS storage as `docker.io/stilldeadcode/radiance`). +`grep -c` receipts: `qwen35moe_fp8.so` carries the `RADIANCE_CALIB` tap +(2) and gram export (2) plus MTP (4) and NO ngram (0) — i.e. the full +calibration→GPTQ path exists for the qwen3.5-MoE family; GPTQ itself +lives in `rad-convert`, not the plugin (0 hits in both `.so`). + +Worked example — `Qwen/Qwen3.6-35B-A3B` (config: `qwen3_5_moe_text`, +40 layers, `full_attention_interval=4` DeltaNet hybrid, **256 experts +top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab +248320, multimodal wrapper with vision tower): +- Expert set ~32 B params → w4 ≈ 16 GiB; trunk (attn/DeltaNet/embed/ + lm_head ~3 B) at i8 ≈ 3 GiB ⇒ **~20 GiB total — fully VRAM-resident + on ONE R9700** (32 GiB card): no TP=2, no host pool, no disk tier, + no `--ngram-placement`, no 47.7 GiB PLE copy, no HC term. The recipe + loses three lines (PLE/HC/protected-count scales with 256 experts). +- Convert cost scales with the file: Flash-Next's 335 GiB took 5 h 50 m + host-backend ⇒ ~70 GiB BF16 ≈ 1–1.5 h. KLD reference boot is trivial + (BF16 ~65 GiB in host pool beside the pool KV needs). +- Unresolved before any real campaign: does `qwen35moe` decode the + `Qwen3_5MoeForConditionalGeneration` vision wrapper (assume text-only + until proven), and does its MTP cast to the fp8/u2 encodings like + qwen4exp. The MNBT-2048 disk-tier law is MOOT for a resident model. + ### Strix Halo — frank-large (halogen) and friends - `.hgn`: **closed production door.** No public quantizer exists (verified absence across peonist-ai repos); the format's lock is the