docs(LLM): Quantization Production reference — per-card self-quant playbooks, REAP/Heretic/abliteration/K_P prior art

serverAppConfigs-docs/LLM/Quantization Production.md

@@ -0,0 +1,161 @@
+# Quantization Production — making our own quants, per card and runtime
+
+**Research page (2026-10-08).** Question: how do WE produce quantizations
+for the RTX 3090, the R9700 (gfx1201), and Strix Halo (gfx1151); what is
+native to each runtime; and the state of the named prior-art families
+(REAP, Heretic, abliteration, HauhauCS K_P "Perfect"). Marked facts:
+**[repo-verified]** = evidenced by a file in this repo; **[verified]** =
+checked first-hand against the cited primary source today; the rest are
+survey claims to re-verify before acting.
+
+## Runtime-native quant matrix
+
+| Card | Runtime | Container | Native weight encodings | Production tooling we own |
+|---|---|---|---|---|
+| RTX 3090 (sm_86) | llama.cpp CUDA | GGUF | K-quants (Q4_K_M…Q8_0), I-quants (IQ1_S…IQ4_XS), ML, f16/bf16; any tensor type per-tensor | `llama-quantize` + `ggml-imatrix` (built by `scripts/build-llama-rdna4.sh` targets) **[repo-verified]** |
+| R9700 (gfx1201) | radiance (StillDeadcode) | `.rad` | per-tensor encodings from a recipe: GPTQ w4 (`w4nl64a8h`), bf16, int8, fp8_e4m3, u2, MXFP4/MXFP6 (vLLM-radiance era) | full pipeline: `rad-convert` + calibration tap + `derive-protected-experts.py` + `kld-gate-swift15.sh` **[repo-verified]** — CPU host-backend, the card is irrelevant to production |
+| R9700 (gfx1201) | vLLM (tcclaviger/R4D) | safetensors | native block-FP8, MXFP4/DSFP4, GPTQ/AWQ, compressed-tensors | GPTQModel/AutoRound are hardware-agnostic torch (CPU-scale); AMD decode kernels are the gate **[survey]** |
+| Strix Halo (gfx1151) | halogen-flash-server (peonist, **closed**) | `.hgn` | vendor-defined (w4b era, v2 checkpoint+ngram+vision files) | **NONE public** — no `.hgn` quantizer exists; every `.hgn` on the fleet was made by peonist/Quat3rnion **[repo-verified gap]** |
+| Strix Halo (gfx1151) | llama.cpp / gufo / Kairic / ROCmFPX forks | GGUF | K/I quants (gufo UD-* = imatrix mixed profiles), custom types: `Q4_0_ROCMFP4_STRIX`, `Q8_0_ROCMFPX` (type 103), Kairic IU4 | we built the RUNTIMES (`scripts/build-kairic-edge-gfx1151.sh`, ROCmFPX pin `36e9acd`), never the custom-type quantizers — those files came from agentionai/kingjones **[repo-verified]** |
+
+Laws that steer everything here:
+- **Production is hardware-agnostic; decode is not.** GPTQ/imatrix/ablation
+  surgery runs in torch/CPU anywhere. The card only picks which ENCODED
+  FORM it can DECODE — hence the one-table-one-encoding law, the GGUF
+  custom-type histogram law ([[GGUF Compatibility]]), and the fact that
+  our w4 convert ran 5 h 50 m on the host backend while the GPUs served
+  nobody (`scripts/convert-swift15-w4dn8.sh`) **[repo-verified]**.
+- **A custom tensor type locks the file to its fork** (ROCmFPX type 103
+  precedent). K_P quants deliberately avoid that trap (below).
+
+## The named families
+
+### REAP — Router-weighted Expert Activation Pruning (Cerebras, 2025)
+- arXiv 2510.13999; `github.com/CerebrasResearch/reap` **[verified]**.
+- One-shot MoE surgery: score every expert by router gate-values ×
+  activation energy, prune the low-utility tail, renumber+export a
+  SMALLER MoE checkpoint (safetensors; GGUF by conversion). Published:
+  GLM-4.5-Air −25 % (82B→~A12B class), GLM-4.7-REAP-268B-A32B-FP8;
+  near-lossless at 25–50 % compression on generative benchmarks, and
+  pruning BEATS expert-merging there **[verified from paper/README]**.
+- Fleet relevance: it is the **inverse of our protected-experts law**.
+  `derive-protected-experts.py` ranks experts by calibration-gram energy
+  share and keeps the hottest 24/layer at bf16; REAP's saliency is the
+  same statistic and deletes the coldest. REAP the Flash-Next MoE (all
+  49,152 experts w4, 512/layer) by 25–50 % → resident set ~58–87 GiB →
+  the expert pool fits with room for KV/cache, killing the disk tier.
+  A REAPed checkpoint re-enters our normal pipeline: stage → calibrate →
+  rad-convert → KLD gate. Hardware-agnostic; runs on the host.
+
+### Heretic (`p-e-w/heretic`) — automated abliteration + layer pruning
+- `github.com/p-e-w/heretic`, AGPLv3 **[verified]**. NOT a quantizer.
+  Fully-automatic censorship removal: directional ablation (refusal
+  direction from PCA of harmful/harmless residuals), per-layer ablation
+  weights, optional LAYER DROPPING (model shrinks), auto-tuned by
+  co-minimizing refusals and KL divergence from the original. Runs on
+  our 3090 (torch, ~24–48 GB working set for 9–27B classes; long batch
+  probes) **[verified README; runtime cost survey]**.
+- Output is a normal HF checkpoint (fp16/bf16) → feeds GGUF, safetensors
+  (vLLM), and `.rad` conversion. Order law: **surgery → calibrate →
+  quantize**; calibrate GPTQ/imatrix on the SURGICAL output, and keep
+  refusal-shaped tokens out of the calibration corpus.
+- Fleet note: our abliterated bases (Quat3rnion v2, HauhauCS
+  Aggressive/Balanced, ukisai Swift1.5) are this family's output; the
+  Quat3rnion repo lore "the abliteration workflow appends, compaction
+  drops dead padding" is exactly Heretic-style layer-editing leaving
+  dead blocks (89.8 → 66.7 GiB r2) **[repo-verified lore]**.
+
+### Ablation / abliteration (directional ablation)
+- Arditi et al. 2024, arXiv 2406.11717: refusal is mediated by a single
+  direction; orthogonalizing weights against it removes refusals with a
+  rank-1 edit. Open implementations: `p-e-w/heretic` (automated, above),
+  Artio/abliteration-toolkits, `EleutherAI/repeng`-style rep-overlap
+  variants **[verified (tool existence: heretic; rest survey)]**.
+- Hardware-agnostic fp-editing. On a MoE the edit touches attention/norm
+  weights, so it composes with any expert quantization.
+- Fleet status: frank-large runs an abliterated base **by Frank's choice**
+  (AGENTS: censored-base refusals were the accepted trade of the 27B era);
+  frank-archon's ukisai Swift1.5 is an abliterated finetune of Qwen3.8.
+
+### HauhauCS QX_K_P "Perfect" quants
+- Our frank-secondary publisher. Method, from the publisher's own cards
+  **[verified]**: (1) abliterate + finetune ("Aggressive"/"Balanced"),
+  (2) quantize with **imatrix calibration on the ABLITERATED weights**,
+  (3) K_P = a **per-model, per-tensor mixed-precision profile** —
+  "model-specific analysis to selectively preserve quality where it
+  matters most", e.g. Q8_K_P = a census of Q8_K + F32 + F16 tensors
+  (FriendliAI's safetensor census of a Q8_K_P shows exactly that
+  399/353/99 tensor split). Result: +1–2 quality rungs for +5–15 %
+  size. **It is a standard GGUF** — llama-quantize with per-tensor type
+  overrides (`--tensor-type`), imatrix on; no fork, no custom types
+  (LM Studio just fails to NAME the mixed file, shows "?").
+- Reproducible with stock tooling: `ggml-imatrix` on the target model +
+  `llama-quantize --tensor-type <pattern>=<type>` per the model's
+  sensitivity census. Both binaries already build from
+  `scripts/build-llama-rdna4.sh` **[repo-verified]**.
+- Our own ladder precedent: the fleet's UD-Q8_K_XL (unsloth "Dynamic" =
+  same family: imatrix + importance-tiered per-tensor types) and
+  agentionai FAST imatrix ROCmFP4-FAST (we ran the imatrix step
+  ourselves on the v2 joined-PLE file — the one self-made quant in repo
+  history) **[repo-verified]**.
+
+## Per-card playbooks
+
+### RTX 3090 — frank-secondary (llama.cpp CUDA)
+1. Surgery on host/3090: abliterate/prune (Heretic) — torch.
+2. `ggml-imatrix` (CUDA build) over a calibration corpus (our
+   `/home/dank/corpora` style set; ≥1M tokens; the fleet's KLD corpus
+   is a reasonable seed).
+3. `llama-quantize` F16 → target rung; K_P-style `--tensor-type`
+   overrides for attention/KV/first-layer upgrades; `--use-repack` for
+   IQ rungs.
+4. Validate: perplexity + KLD vs BF16 + the K-quant rank checks.
+Decode-side options we do NOT use: EXL3 (turboderp ExLlamaV3, QTIP
+trellis; CUDA production on the 3090 works, decode is exllama-only —
+new runtime, no OpenAI server), Marlin-GPTQ (vLLM path, not a llama.cpp
+runtime).
+
+### R9700 — frank-primary (radiance)
+The full self-quant pipeline already exists end to end: fetch BF16 →
+record KLD reference (1.0.13) → capture calibration grams (1.2.0 tap,
+`RADIANCE_CALIB_DIR/TOKENS`) → derive protected experts (energy-share) →
+`convert-swift15-w4dn8.sh` (1.0.13, host backend) → KLD gate (MNBT 2048
+law, pool 40960, mean KL ≤ 0.03, top-1 ≥ 95 %). New model = new recipe
++ new `.so` arch plugin; every step is scripted. Prior art to fold in:
+REAP the expert set BEFORE step 3; ablate BEFORE calibration.
+Native encodings beyond w4: i8 trunk, fp8 HC/PLE/KV, bf16 protected,
+u2 draft — all recipe-expressible (see the w4dn8 runbook).
+
+### Strix Halo — frank-large (halogen) and friends
+- `.hgn`: **closed production door.** No public quantizer exists
+  (verified absence across peonist-ai repos); the format's lock is the
+  engine's lock. We consume vendor artifacts; the repo-side tooling is
+  compaction/verification only.
+- Self-quant work on this card therefore runs through **GGUF runtimes**:
+  the same imatrix/llama-quantize flow as the 3090 (HIP/Vulkan decode of
+  K/I quants, gufo UD-* precedent, `--use-repack` off on ROCm), or the
+  custom-type forks (ROCmFP4 STRIX, Kairic IU4) whose PRODUCERS we have
+  never been — the quantizers for types 101/103 ship in those forks'
+  trees; building one is a fork build, not a flag.
+- Native-FP4 note: RDNA4/Strix get hardware FP4 matmuls (hipBLASLt
+  MXFP4/DSFP4), which is the engine-level path (radiance/vLLM), not a
+  GGUF decode.
+
+## Sources
+- REAP: arXiv 2510.13999, github.com/CerebrasResearch/reap,
+  cerebras/GLM-4.5-Air-REAP-82B-A12B-FP8, cerebras/GLM-4.7-REAP-268B-A32B-FP8.
+- Heretic: github.com/p-e-w/heretic (AGPLv3).
+- Abliteration: arXiv 2406.11717 (Arditi et al.).
+- K_P: HauhauCS model cards (e.g. `HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive`
+  README "What are K_P quants?"), FriendliAI Q8_K_P tensor census,
+  fleet payload tree (BF16/Q4_K_M/Q6_K/Q8_0 + mmproj ladder in
+  `HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive`).
+- EXL3: github.com/turboderp-org/exllamav3 (QTIP lineage; CUDA).
+- Repo: `scripts/convert-swift15-w4dn8.sh`,
+  `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
+  `scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
+  `scripts/build-kairic-edge-gfx1151.sh`, [[agentionai-fast-imatrix]],
+  [[kingjones-strix-rocmfp4]], [[qwen3.8-strix-optimizations]],
+  [[frank-primary-radiance-swift15-w4dn8]],
+  [[frank-secondary-llama-qwen35-9b-q6k-2x200704]],
+  [[frank-large-flash-server-v2-swift15]], [[GGUF Compatibility]].

serverAppConfigs-docs/LLM/index.md

@@ -492,6 +492,9 @@ frank-secondary, port 7771]
 One polished reference page per skill-cluster, distilled from the managed
 skill library into the vault (see [log](../log.md) for the wave plan):
 
+- [[Quantization Production]] — how we self-make quants per card
+  (imatrix/K_P ladders, radiance recipes, REAP/Heretic/abliteration
+  prior art), and what each runtime decodes natively
 - [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand
   of a loader; how the runtime is chosen by the file
 - [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape

serverAppConfigs-docs/index.md

@@ -159,7 +159,7 @@ skill-cluster. See [log.md](log.md) for the wave plan and results.
   [[vLLM Porting and Compile Triage]] · [[LLM Bench Windows and Measurement]] ·
   [[LLM Performance Diagnosis]] · [[Model Swap and Rollback]] ·
   [[Tcclaviger Fork Ops]] · [[Heterogeneous Split Triage]] ·
-  [[FreeToken Builds]]
+  [[FreeToken Builds]] · [[Quantization Production]]
 
 **Agent:**